Normalization and cumulative analysis of cognitive educational outcome elements and related interactive report summaries

ABSTRACT

The systems, methods and associated devices electronically collect, report and generate normalized educational outcome summaries of multiple different educational inputs, including didactic, experiential and problem solving events and/or assessments.

RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. application Ser. No. 13/425,627 filed Mar. 21, 2012, whichclaims the benefit, under 35 U.S.C. § 119(e), of U.S. ProvisionalApplication Ser. No. 61/466,207, filed Mar. 22, 2011, entitled“NORMALIZATION AND CUMULATIVE ANALYSIS OF COGNITIVE EDUCATIONAL OUTCOMEELEMENTS AND RELATED INTERACTIVE REPORT SUMMARIES”, the disclosures ofwhich are incorporated herein in their entireties by reference.

COPYRIGHT PROTECTED MATERIAL

A portion, of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner, EastCarolina University of Greenville, N.C., has no objection to thereproduction by anyone of the patent document or the patent disclosure,as it appears in the Patent and Trademark Office patent file or records,but otherwise reserves all copyright rights whatsoever.

FIELD OF THE INVENTION

The invention relates to educational assessment systems that canevaluate student competency in different sub-topics related to aneducational topic, concept or field and/or evaluate cohort factorsassociated with positive and negative cognitive test outcomes.

BACKGROUND

In the past, educational and testing systems were designed to evaluatestudents based primarily on didactic based tests. To automate suchevaluations, the use of microcompetency codes for curriculum mapping ofdidactic topics has been proposed, Others have used MeSH (MedicalSubject Heading) codes for certain types of educational evaluations ofmedical curriculums. However, these codes have not resulted in apractical way to evaluate students, particularly in a manner that canconsider other types of input. Indeed, such prior approaches are onlyable to evaluate students for topics presented in didactic environments.

Many educational programs should require proof of knowledge, skills, andinter-disciplinary problem solving. However, it is difficult to assessthese different educational outcomes longitudinally. For example, indental and medical education, and other educational fields, it isbelieved that a greater degree of student competencies should be basedon other factors, e.g., over 60% of the student competence should bemeasured in clinical environments.

There remains a need for alternate evaluation systems that can provideimproved competency-based assessments.

SUMMARY

Some embodiments of the present invention are directed to methods forproviding data for evaluating student competency. Such methods mayinclude generating an evaluation grid for at least one student. The gridmay include multiple different microcompetencies, multiple scores thatare associated with corresponding ones of the differentmicrocompetencies, the scores corresponding to at least one didacticevent, at least one experiential event, and at least one, discussionevent. Some embodiments provide that generating the evaluation grid isperformed using at least one computer processor.

In some embodiments, ones of the scores for corresponding ones of thedifferent microcompetencies are relative educational value (RVU) scoresand each of the at least one didactic event, the at least oneexperiential event and the at least one discussion event that is used togenerate a respective score is associated with a metadata codeidentifying a topic code corresponding to ones of the differentmicrocompetencies and RVU scores. In some embodiments, generating theevaluation grid is performed using the metadata codes.

Some embodiments further include accumulating RVU scores for differentdidactic events, experiential events and discussion events, correlatedto respective students over time and updating the evaluation grid basedon the accumulated RVU scores. In some embodiments, the grid is updatedat a substantially regular periodic interval. Some embodiments providethat the substantially regular periodic interval is at least a weeklyinterval to reflect changes in student scores corresponding to ones ofthe plurality of different microcompetencies.

In some embodiments, the RVU scores from each event are time-normalizedscores and the didactic and experiential RVU scores are based on binarycharacterizations of test and experience events. Some embodimentsprovide that the experiential environment RVU scores are based on apre-defined assessment of difficulty and an estimated time to complete arespective experiential task and the experiential task is associatedwith more than one topic code corresponding to ones of the differentmicrocompetencies.

Some embodiments provide that the discussion environment RVU scores arebased on user-defined RVU scores for a student that are assigned afterevaluating a student online discussion.

In some embodiments, the grid is an interactive grid. Some embodimentsfurther include allowing a user to select a cell in the grid to revealunderlying supporting data of a respective microcompetency and/orstudent.

Some embodiments include, displaying the grid with cells in a respectivemicrocompetency having a color that is associated with a defined status.In some embodiments, the defined status corresponds to a relativeperformance of the student among a plurality of other students in aplurality of the students that includes the student. Some embodimentsprovide that the relative performance is based on a standard deviationof the RVU scores for the plurality of students and cells in the grid ina respective microcompetency are displayed using a first color thatcorresponds to a score identified as being below a statistically definedminimum, a second color that corresponds to a score that is above thestatistically defined minimum and below a statistically definedexcellence threshold, and a third color that corresponds to a score thatis above the statistically defined excellence threshold. In someembodiments, cells in the grid in a respective microcompetency aredisplayed using the first color that corresponds to a score identifiedas being below a non-statistically defined minimum exclusive of thestatistically defined minimum. Some embodiments provide that cells inthe grid in a respective microcompetency are displayed using the thirdcolor that corresponds to a score that is above a non-statisticallydefined excellence threshold exclusive of the statistically definedexcellence threshold.

Some embodiments of the present invention include methods of providingdata for evaluating a student's competency in a topic. Such methods mayinclude obtaining relative educational value unit (RVU) scores fordifferent defined microcompetencies by electronically identifyingassociated ones of a plurality of metadata codes for a plurality ofdifferent microcompetencies that are correlated to student identifiersfrom didactic, experiential and discussion environments over time.Methods may also include storing the obtained RVU scores in associationwith supporting reports.

Some embodiments include generating a cumulative analysis grid based onthe RVU scores. In some embodiments, generating the cumulative analysisgrid includes mathematically summing RVU scores from each of thedidactic, experiential and discussion environments for respective onesof the plurality of different microcompetencies and updating thecumulative analysis grid based on subsequently obtained cumulative datafor respective students.

In some embodiments, didactic RVU scores are based on binarycharacterizations of test events and experiential RVU scores are basedon binary characterizations of experiential events.

Some embodiments provide that obtaining relative educational value unit(RVU) scores includes receiving an exam data file that corresponds toeach didactic event, the exam data file including a unique studentidentifier, a test item identifier, a microcompetency code correspondingto the test item and a binary answer choice value. Some embodimentsinclude modifying the received exam data file to include at least one ofa program identifier, an exam date and a course identifier and storingthe modified exam data file.

Some embodiments further include programmatically validating themodified exam data file by comparing contents therein with contents ofthe exam data file. Some embodiments further include displaying contentof the exam data file for validation by a user.

Some embodiments further include receiving a commitment input and,responsive to receiving the commitment input, converting data from themodified exam data file into summary data correlated by microcompetencyto provide topic-associated results. Some embodiments further includereceiving a validation input that indicates that the summary datacorrelated by microcompetency is approved and, responsive to receivingthe validation input, generating aggregate data that associates RVUscores corresponding to the summary data with corresponding students.Some embodiments include receiving a commitment input that indicatesthat the aggregate data is approved, tagging a file corresponding to theaggregate data, the summary data and/or the modified exam data ascommitted, and updating a cumulative analysis grid based on RVU scoresin the aggregate data.

Some embodiments of the present invention include one or more circuitsconfigured to generate an interactive cumulative grid of a plurality ofdefined educational topics associated with a cognitive competency of astudent.

Some embodiments of the present invention include computer programproducts for providing competency-based student evaluations. Suchcomputer program products may include a non-transitory computer readablestorage medium having computer readable program code embodied in themedium. In some embodiments, the computer-readable program code includescomputer readable program code that generates a summative grading outputbased on an evaluation of didactic test events associated with definedassociated microcompetency topic codes and relative educational valueunits. Embodiments may include computer readable program code thatgenerates a summative grading output based on an evaluation ofexperiential individual experience elements associated with definedassociated microcompetency topic codes and relative educational valueunits and computer readable program code that generates a summativegrading output based on an evaluation of individual discussion eventsassociated with defined associated microcompetency topic codes andrelative educational value units. Embodiments may further includecomputer readable program code that generates a cumulative analysisstudent evaluation grid using the summative grading outputs.

In some embodiments, the cumulative analysis student evaluation gridusing the summative grading outputs includes cells in the grid that aredisplayed in a respective microcompetency having a color that isassociated with a defined status. Some embodiments provide that thedefined status corresponds to a relative performance of the studentamong a plurality of other students in a plurality of the students thatincludes the student based on a standard deviation of the relativeeducational value units for the plurality of students, anon-statistically defined minimum that is defined independent of theplurality of other students and a non-statistically defined excellencethreshold that is defined independent of the plurality of otherstudents.

Some embodiments of the present invention include educational analysissystems that include at least one web-based service with at least oneserver that is configured to accept electronic input fromprofessors/teachers and students to communicate with the web-basedservice to interactively participate in timed discussion events withstudents and student groups, and wherein the system is configured toprovide an input window to allow professors/teacher to inputmicrocompetency codes and relative educational value unit scores for arespective discussion event for each student and each student groupparticipating in the discussion event.

In some embodiments, the at least one web-based service is furtherconfigured to use metadata codes to relate defined individualexperiential events with an associated one of the microcompetency codesand at least one of the relative educational value units.

As will be appreciated by those of skill in the art in light of theabove discussion, the present invention may be embodied as methods,systems and/or computer program products or combinations of same. Inaddition, it is noted that aspects of the invention described withrespect to one embodiment, may be incorporated in a different embodimentalthough not specifically described relative thereto. That is, allembodiments and/or features of any embodiment can be combined in any wayand/or combination. Applicant reserves the right to change anyoriginally filed claim or file any new claim accordingly, including theright to be able to amend any originally filed claim to depend fromand/or incorporate any feature of any other claim although notoriginally claimed in that manner. These and other objects and/oraspects of the present invention are explained in detail in thespecification set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a schematic illustration of an evaluation system/methodaccording to some embodiments of the present invention.

FIG. 2 is a schematic illustration system/method similar to that shownin FIG. 1 with an additional analysis platform according to someembodiments of the present invention.

FIG. 3 is a block diagram illustrating a display screen in a graphicaluser interface of an exemplary discussion event input according to someembodiments of the present invention.

FIG. 4 is a block diagram illustrating a display screen of an exemplarydiscussion event input according to some embodiments of the presentinvention.

FIG. 5 is a screen shot of an example of a discussion thread andassociated CSV file according to some embodiments of the presentinvention.

FIG. 6 is a partial screen shot of an exemplary interactive(color-coded) evaluation grid according to some embodiments of thepresent invention.

FIGS. 7A and 7B are competency grid listings of two respective sets ofassociated microcompetencies according to some embodiments of thepresent invention.

FIG. 8 is a schematic illustration of a dashboard with restricted viewsbased on user profiles/types according to embodiments of the presentinvention.

FIG. 9 is a flow chart of exemplary operations that can be performedaccording to some embodiments of the present invention.

FIG. 10 is a schematic illustration of one example of a web-based systemaccording to some embodiments of the present invention.

FIG. 11 is a block diagram of a data processing system according to someembodiments of the present invention.

FIG. 12 is a screen shot of a graphical user interface for a sub-cohortmanager according to some embodiments of the present invention.

FIG. 13 is a screen shot of a graphical user interface for a summativereport after grading according to some embodiments of the presentinvention.

FIG. 14 is a screen shot of a graphical user interface for a postverification report before the data is submitted to the grid accordingto some embodiments of the present invention.

FIG. 15 is a screen shot of a graphical user interface for a managingsubmitted reports according to some embodiments of the presentinvention.

FIG. 16 is a partial screen shot of an exemplary interactive evaluationgrid that is parsed to display a single anatomical system according tosome embodiments of the present invention.

FIG. 17 is a partial screen shot of a graphical user interface for amanaging an interactive evaluation grid where multiple grids arepresented for editing according to some embodiments of the presentinvention.

FIG. 18 is a partial screen shot of an exemplary interactive evaluationgrid that is parsed to analyze the data by a single discipline accordingto some embodiments of the present invention.

FIG. 19 is a partial screen shot of a graphical user interface includinga component of an interactive evaluation grid that allows selectiveviewing of one or more modalities and for the definition of an analysisdate range according to some embodiments of the present invention.

FIG. 20 is a screen shot of a graphical user interface of a cohortmanager that determines which students and faculty are included in aparticular cohort according to some embodiments of the presentinvention.

FIG. 21 is a screen shot of a graphical user interface for a managing aninteractive evaluation grid where competencies include microcompetencycodes according to some embodiments of the present invention.

FIG. 22 is a screen shot of a graphical user interface illustrating rawimported exam data for a single student after an item analysis has beenperformed according to some embodiments of the present invention.

FIG. 23 is a screen shot of a graphical user interface for verifying araw data report in preparation for validation according to someembodiments of the present invention.

FIG. 24 is a screen shot of a graphical user interface illustrating anRVU Commit Summary screen before data is committed according to someembodiments of the present invention.

FIG. 25 is a screen shot of a graphical user interface illustrating datathat was collected and merged by microcompetency code for differentstudents according to some embodiments of the present invention.

FIG. 26 is a screen shot of a graphical user interface illustrating datathat was collected and merged by microcompetency code and that isverified to provide all students with the correct score according tosome embodiments of the present invention.

FIG. 27 is a screen shot of a graphical user interface illustrating anRVU Commit Summary screen that includes RVU scores ready to commit toone or more data bases for the grid according to some embodiments of thepresent invention.

DETAILED DESCRIPTION

The present invention will now be described more fully hereinafter withreference to the accompanying figures, in which preferred embodiments ofthe invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein.

Like numbers refer to like elements throughout. In the figures, layers,regions, or components may be exaggerated for clarity. Broken linesillustrate optional features or operations unless specified otherwise.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items. As used herein, phrases such as “between X and Y” and“between about X and Y” should be interpreted to include X and Y. Asused herein, phrases such as “between about X and Y” mean “between aboutX and about Y.” As used herein, phrases such as “from about X to Y” mean“from about X to about Y.”

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the specification andrelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein. Well-known functions orconstructions may not be described in detail for brevity and/or clarity.

It will be understood that when an element is referred to as being “on”,“attached” to, “connected” to, “coupled” with, “contacting”, etc.,another element, it can be directly on, attached to, connected to,coupled with or contacting the other element or intervening elements mayalso be present, in contrast, when an element is referred to as being,for example, “directly on”, “directly attached” to, “directly connected”to, “directly coupled” with or “directly contacting” another element,there are no intervening elements present. It will also be appreciatedby those of skill in the art that references to a structure or featurethat is disposed “adjacent” another feature may have portions thatoverlap or underlie the adjacent feature.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,features, steps, layers and/or sections, these elements, components,features, steps, regions, layers and/or sections should not be limitedby these terms. These terms are only used to distinguish one element,component, feature, step, region, layer or section from another region,layer or section. Thus, a first element, component, region, layer,feature, step or section discussed below could be termed a secondelement, component, region, layer, feature, step or section withoutdeparting from the teachings of the present invention. The sequence ofoperations (or steps) is not limited to the order presented in theclaims or figures unless specifically indicated otherwise.

The term “student” refers to the individual(s) being evaluated. As usedherein, the term “substantially real time” includes receiving and/ortransmitting data between sites during a discussion or test accountingfor system delays in remote transmission between sites which may be onthe order of seconds or less or potentially minutes in length as aresult of routing, traffic, transmission route and/or systemcommunication link employed which can impede the transfer such thatslight delays may occur.

The term “automatic” means that substantially all or all of theoperations so described can be carried out without requiring theassistance and/or manual input of a human operator. The term“electronic” means that the system, operation or device can communicateusing any suitable electronic media and typically employsprogrammatically controlling the communication between participantsusing a computer network.

The term “programmatically” means the action is directed via a computerprogram code.

The term “hub” means a node and/or control site (or sites) that controlsand/or hosts data exchange between different user sites using a computernetwork. The term “FERPA” refers to the Family Educational Rights andPrivacy Act (FERPA) under the laws of the United States of America.

The term “formative evaluation” refers to a cross-sectional event whereindividuals are provided guidance to improve performance. Formativeevaluations are not required to be submitted for grades or points.

The term “summative evaluation” refers to a cross-sectional event whereindividuals are given an evaluation of performance in the form of pointsor grades.

The term “outcome element” is a unit of defined educational outcome aspart of a summative evaluation. An outcome element may contain thefollowing data: unique identifier, student unique identifier, summativeassessment identifier, evaluation type, and/or success/fail. Someembodiments provide that success/fail may be a binary value. Whennormalized, an outcome element may contain mircocompetency code(s) andrelative educational value unit(s). The normalized element may ignorefailed items and may only give credit for successful items.

The term “cumulative evaluation” refers to a parsed collection of many,typically all, summative evaluations to provide a balanced, if notcomplete, analysis of student performance in a program, curriculumand/or competency.

The term “cumulative analysis engine” refers to an electronic circuitthat can electronically analyze student summative evaluation data inputsover time from at least one of the different educational evaluationenvironments, and typically three or more education evaluationenvironments (e.g., didactic, experiential, and discussion, amongothers) to generate a representation of the cohort performance relativeto competency definitions.

Educational evaluation of performance may use formative and summativeevaluations relative to a set of curriculum standards. Some embodimentsprovide that this can be done with basic scores on exams that areaveraged to give a grade. In complex outcomes environments, simplegrades may not provide enough information to correct specificdeficiencies. The term “competency” has been used to give a more precisedefinition of skills and knowledge required to perform integrated tasks,such as those corresponding to medicine, engineering, and/or law, amongothers. Some embodiments may be particularly suitable for competencyevaluation of students using normalized inputs associated with didacticgrading, experiential grading and discussion environment grading. Someembodiments of the present invention can be used to assess effectivenessof complex curricula based on student competency scores. Any competencycan be represented with a statement and/or definition and/or can berepresented by multiple sub-components. As used herein, the sub-topicscorresponding to competencies, may be referred to as “microcompetencies”and will be detailed in later sections.

In some embodiments, the systems/methods are not configured to defineobjective differences between relative skills. The student encountersmay be generally, and more typically, totally binary. As such, adidactic environment may be limited to reporting elements that arecorrectly answered (but may discard or give credit for knownfalse-negatives). An experiential (e.g., clinical) environment mayreport skills that are successfully completed. A discussion environmentmay report posts that are made therein. These reports are defined ascognitive evaluations, meaning that relative quality of individualmeasures may not be used. Non-cognitive subjective values may beevaluated with other techniques.

In competency-based curricula there may be three different modalitiesfor delivering instruction, and therefore, three different environmentsfor evaluation. According to some embodiments, the three differentenvironments for evaluation can be integrated into an extensiblecompetencies electronic evaluation platform. The environments forevaluation may be the didactic modality, the experiential modality, andthe discussion modality. Each will be detailed in separate sections ofthis document. Not all educational fields use all modalities forevaluation.

“Normalization” and the term “normalized” refer to a defined correlationstandard for measuring different factors. For example, a point valueassigned to individual tests or test questions, time spent in clinicalor on experiential tasks, values assigned to critical thinking exercisesand the like. According to some embodiments, summative reports from manydifferent environments may be digested when available. The results inthe individual digested summative reports may be normalized into acommon format. All of the summative reports may be combined into acommon data set and a graphical representation of the data may beprovided in a Cumulative Analysis Grid. Some embodiments provide thatnormalization may include defining relative value of points based oneducational time spent, categorizing possible topics taught(microcompetency) and defining individuals, sub-cohorts of individuals,and cohorts of individuals that may be cumulatively evaluated.

The term “minimal time of relative value” (T) may refer to the basisunit for assigning credit for any outcomes event. T is the base time andcan be further multiplied by other factors to assign event credit. Thedifferent inputs may be normalized using the minimal time of relativevalue (T). For example, a test question may have numeric output of “1”or “2” based on how long it is predicted that a typical student may needto evaluate and answer the question. The number can be assigned in timeequivalents defined by a particular institution and/or based on astandard, such as a “1” for a 15 minute equivalent (thus a “2” can beassociated with questions rated at a “30 minute” projected responsetime). If a student correctly answers the question, the answer receivesthe defined (normalized score). For experiences, like clinical practiceof medicine or dentistry, the actual time that it takes to perform thetask and the relative complexity of the task can be given a normalizedvalue relative to 15 minutes. For example, the average time to extract atooth may be 15 minutes, therefore, 1 value unit is given. This makesgetting one question on a didactic exam equal to taking out a tooth inclinic, front a normalized basis. Similarly, for discussion-basedenvironments, it is possible to give partial points for individualresponses to problem-based learning discussions.

The term “relative educational value unit” (RVU) is an expression ofpossible credit for a skills assessment that is relative to time andcomplexity. For example, assuming that a didactic test item is equal to1 RVU, then an experiential procedure or discussion experience may bemapped to additional time or complexity. A RVU can be expressed as equalto a basic time interval (T), multiplied by increments of T when morethan one time interval is associated with a task or discussion.Additionally, the RVU may be adjusted for complexity: T×n (multiple ofT)×C (complexity multiplier). The RVU may provide a relative value scoreassociated with different educational factors, e.g., test scores, testanswers, clinical or other experiential tasks or hours, discussion basedproblem solving skill scores and the like, normalized to a time unit(T). Thus, grading using RVUs is time-equivalent normalized and may beused for summative outcome evaluation of different categories oftesting/evaluation.

The term “microcompetency codes” corresponds to microcompetencies andrefers to a hierarchical expression of different topics that arepossible for a student to experience during a competency-basededucational curriculum. For example, a plurality of microcompetenciesare associated with an overall competency for a particular curriculum.Not all codes are necessarily expressed in the curriculum, but themicrocompetencies are a superset of what is possible to, encounter.Additionally, although generally used herein in the plural form“microcompetencies”, the singular form “microcompetency” is includedtherein.

The term “topic” refers to a defined educational concept, field orsubject. The term “topic code” refers to an identifier that iscorrelated to a defined topic. The term “identifier” refers to a uniqueset of characters, typically numeric and/or alphanumeric characters. Theidentifier may be in a defined format for standardization acrossmultiple electronic evaluation platforms. The microcompetencies areexpressed as a defined hierarchical set of individual microcompetenciesthat correspond to a sub-topic of one or more defined topic codes. Thatis, one microcompetency may be associated with more than one topic code.A topic code typically includes a plurality of associatedmicrocompetencies and may include between about 10-100 for somecompetency topics, although such range is not limiting. For example,some embodiments herein provide that more or less microcompetencies maybe associated with different topics and different topics may havedifferent numbers of microcompetencies. The microcompetencies may beprovided with sufficient numbers to allow granular evaluation of adifferent sub-concepts within a particular topic. The microcompetenciesmay themselves be related to a plurality of sub-microcompetency codes. Aparticular educational assessment system may have several hundred topiccodes and thousands of microcompetencies. The microcompetencies may beuniquely coded, for example, with a numerical code, but other code typesmay be used. The code may identify the relationship and/or position of atopic within the hierarchy. “Competency” may be defined as a combinationof different microcompetency codes. Further, the same microcompetencymay appear in multiple different competency evaluations.

The term “cohort” refers to a group of students who are being evaluatedusing the same identified components, elements or factors and/or thecompetencies, and/or microcompetencies. Some examples of cohorts mayinclude students grouped by a class, a professor, an associatedinstitution (e.g., college or graduate school), and/or an assignededucational resource for a class (e.g., a metacoded book), among others.Additionally, the discussion modality may warrant another groupingfactor that may be addressed by defining sub-cohorts. For example,subsets of students can be assigned to small groups for encounteringdiscussions. Additionally, some embodiments provide that each individualin the cohort is also a member of their own sub-cohort. The details ofthis process are explained in the discussion modality section.

The term “didactic modality” refers to evaluation of student learningbased on classroom lectures, textbooks and homework.

The term “question element” (QE) refers to a single didactic-modalityquestion that includes a stem, one or more distractor answers, onecorrect answer or combination of choices, and that has a relative valueequal to T. For example, a question element may include a simple examquestion.

The term “question group” (qgroup) is an expression of a collection ofindividual didactic questions under a common microcompetency. Thedidactic-modality questions can be mapped to the microcompetencies atthe point of creation and/or at the submission to the system as atesting event report.

The term “testing event” (TE) is a combination of question elements tocreate a formative evaluation of the cohort or a sub-cohort. Note thatthe didactic evaluation of sub-cohorts can lead to non-representativeresults in cumulative analysis.

The term “testing event report” (TER) expresses a cohort performance ona testing event.

The term “item analysis” involves mathematical evaluation of the TER toidentify TEs that should be removed due to poor question construction orpoor student performance. These tools may vary widely depending on theinstitution and/or testing mechanism. In some embodiments, the itemanalysis involves evaluating the individual relative to the cohort.

The term “didactic modality summative report” (DMSR) is a list of eachindividual's performance on each FE within the cohort after the itemanalysis have been accomplished and specific TEs have been eliminatedfrom the TER.

The term “didactic modality normalized summative report” is a list ofeach individual's performance from the DMSR aggregated by RVUs bymicrocompetency. This report may be verified against the DMSR and thenmay be submitted to the cumulative data storage for analysis by thevarious analysis grids.

The term “experiential modality” refers to clinical and/or other“hands-on” type experiences related to a microcompetency code.

The term “procedure anchor code” (PAC) is the expression of codedprocedures that can be competed for skills assessment in a clinicalsetting. In the case of health science they are the ICD-10 codes formedical procedures and CDT codes for dental procedures. In practice,some embodiments provide that the procedure anchor codes are representedand may be later mapped to a subset of microcompetencies.

The term “experience element” (EE) refers to an individual performing anactual skill-related task.

The term “experience group” (EGroup) is an expression of a collection ofprocedures/experiences under a common microcompetency. The experiencemodality procedures can be mapped to one or more microcompetencies atthe point of creation and/or at the submission to the system as anexperience event report. An “experience event report” (EER) expressescohort performance on a testing event during a specific range of timeand/or predefined time interval. For example, some embodiments providethat experience event reports may include daily reports from one or moreother systems.

The term “experience modality normalized summative report” (EMSR) is alist of each individual's performance from the EMSR aggregated by RVUsby microcompetency. The PAC may be replaced with a microcompetency andits related RVU. This report may be verified against the EMSR and thensubmitted to the cumulative data storage for analysis by the variousanalysis grids.

The term “discussion modality” refers to a problem solving or discussionforum related to a microcompetency code where a student's ability tosolve a defined problem and/or provide a detailed discussion of adefined discussion element demonstrating proficiency and/orcomprehension and critical thinking is able to be given a grade. Thediscussion modality may be an online environment, a paper-basedenvironment and/or may be a classroom environment. Some embodimentsprovide that the discussion modality is provided in an online formatthat accepts user responses and can be (interactively) monitored and/orassessed by a teacher, professor, teacher assistant, and/or othereducational evaluator. To promote test integrity, a camera mode can beused and/or biometric inputs can be used to validate that the responderis the student. In other embodiments, a dedicated test site for thetesting can be used mid student identify can be validated upon access tothe site.

An “individual discussion event” (IDE) is an individual making a commentin a discussion environment. There are many different types of IDEs andtheir value, may be different for different types and/or content ofcomments.

As used herein, a “discussion sub-cohort” is a subset of the totalcohort that facilitates the discussion educational experiences. In someembodiments, the entire cohort can be a discussion sub-cohort and/or anindividual can be a discussion sub-cohort. Some example embodimentsprovide that a typical discussion sub-cohort is 5-10 individuals,however, such example is non-limiting in that sub cohorts may be lessthan 5 or more than 10 individuals.

A “discussion sub-cohort summative report” (DSSR) expresses discussionsub-cohort performance on a discussion event during a specific range oftime and/or time interval. In some embodiments, DSSRs may include aweekly reports from one or more other systems. A DSSR may be sentdirectly to the system for assignment of microcompetencies and RVU.Unlike the other two environments the topics discussed may be limited totagging after they happen.

A “discussion sub-cohort normalized summative report” is a list of eachindividual's performance from the DSSR aggregated by RVUs bymicrocompetency. A decision may be made within this report as to themembers of the sub-cohort who will receive credit for the discussions.The entire group can be given equal credit as a team, or the individualscan be given credit individually. The normalized data may be submittedto the cumulative data storage for analysis by the various analysisgrids.

As used herein, the term “cumulative grid” (also referred to as the“grid”) refers to a summary of competency related scores of (e.g.,microcompetencies, groups of microcompetencies, and/or OEs) for one ormore students. The grid can be color-coded to show degrees of competencesuch as whether a student meets defined threshold levels of competenciesin different microcompetencies and/or sub-microcompetencies. The gridcan be interactive and/or parseable to allow a user to access supportingdata associated with the reported scores which can electronicallyorganize the student data in various ways to analyze positive andnegative trends associated with different classes, students and groupsof students as well as common factors with different students.

The term “interactive grid” refers to a grid that includes elements thatcan be selected by a user (via a UI such as a GUI) to thenelectronically reveal underlying data supporting that element. Thus,when several students are identified as below minimum for a definedcompetency (e.g., a competency defined by one microcompetency, groups ofmicrocompetencies, one or more sub-microcompetencies and/or groups ofsub-microcompetencies for a topic), some embodiments disclosed hereinmay allow a user to access and/or interrogate (e.g., point and click ona block) the grid to identify individual ratings in varioussub-competencies, as well as common factors, such as professor, school,class time, textbook, (clinical) experiences or tasks, and/or a locationwhere the tasks were performed, among others. This cohort or associateddata can allow educators or schools to adjust curriculums to addressstudent needs where deficiencies are identified, for example.

The term “registered” means that the user is a recognized onlineparticipant of the system, typically using a password and login and/orauthorized portal. The term “administrative user” refers to a user thatdoes not have permission to access student records. Different types ofadministrative users can have different access levels to the system.Some participants/users may have access to cohort data correlated tostudent success, without any student identifiers. The term “web-based”means that the service uses at least one server to communicate withdifferent users over one or more networks including, for example, theWorld Wide Web (e.g., the Internet), using, for example, the hypertexttransfer protocol (HTTP), among others.

Embodiments or aspects of the present invention may be carried out usingany social network service, typically a social network service that canprovide suitable security or private (or semi-private) communications.The term “Yammer®” refers to an enterprise social network service thatwas launched in September 2008 by Yammer, Inc., San Francisco, Calif.Unlike Twitter®, which is known for broadcasting messages to the public,Yammer® is used for private communication within organizations orbetween organizational members and pre-designated groups, making it anexample of enterprise social software. It is contemplated that othersuitable enterprise social software/systems/services may be used tocarry out aspects of the present invention.

As shown in FIGS. 1 and 2, embodiments of the invention include systemsand methods of analysis 10 that include data from one, two or all threedifferent environments: didactic modality 20, experiential modality 40and discussion modality 80. Each modality 20, 40 and 80 is able togenerate respective summative evaluations 28, 48, 88, which areassociated with the metadata codes 15 including, microcompetencies topiccodes 16, RVUs 17, individual (student specific) codes and cohort (e.g.,class, professor, book, learning institution, etc.,) codes 18. The dataunderlying each report and/or outcome element can be electronicallystored for ease of future retrieval as evidence of performance and/orfor curriculum or other evaluation.

Although not limited thereto, it is contemplated that some embodimentsdescribed herein may also be used in conjunction with a licensing systemsuch as for state legal bar examinations for lawyers, and/or licensureexaminations for doctors or veterinarians, among others.

It is also contemplated that some embodiments disclosed herein canevaluate cumulative outcome data with its rich underlying cohort data toprovide feedback to educational institutions, book publishers, and thelike based on pass rates, topic specific achievements for variousmicrocompetencies, and the like over multiple students and in multiplestates. This data analysis can allow such institutions, or otherorganizations to rank schools, rank professors and/or classes, evaluatetext books (noting those that provide the best student outcomes for aparticular topic and/or those that produce poor results), rewardbest-outcome educators for one or more topics, and/or make other changesto a class or curriculum based on such cohort data and cognitive outcomeresults.

It is also contemplated that the systems/methods can be used to evaluatecontinuation education CLE evaluation may only evaluate one or a sub-setof the three environments. First, the didactic environment provides fora direct delivery of content and a relatively simple assessment usingquestions and answers. Second, the clinical environment provides for theperformance of skills and a relatively simple assessment of skillperformance. Third, the discussion environment provides for delivery ofstated scenarios that require research and synthesis and a relativelycomplex assessment of problem-solving behaviors and skills.

Didactic Modality

Still referring to FIG. 1, the didactic modality 20 can include outcomeelements 28 that are based on individual testing elements (block 21)that are electronically (pre)tagged with an associated microcompetencyand RVU (block 22), then grouped into examinations for respectiveindividual testing element summative evaluation (block 23). The groupingcan group sets of individual testing elements 21 with scores (normalizedusing RVUs) (block 23) for summative evaluations×N (block 24). Thus, thestrident encounter with the examinations may be filtered throughextensible outcome element item analysis as a binary evaluation (block25), with post-item analysis summary report (block 26) before beingsubmitted and/or used for outcome element summative grading 28 and datastorage 90, and electronic cumulative analysis 100.

Didactic modality 20 may provide summative evaluations 28 based onindividual test elements that are electronically pre-tagged with one ormore associated microcompetencies and a corresponding RVU. Stateddifferently, exam questions are associated with topics and points, thencollected into an exam for students to test their knowledge.

For many educational environments, individuals are evaluated forknowledge based on simple questions with single correct responses. Thesequestions are typically given in collections as tests and exams.Performance may be based on relative percentage of correct responses.Thresholds for summative analysis may be relatively simple. Examples ofassociated steps for this evaluation are described below:

Step 1. Question stems are associated with responses, which are taggedas correct or as distractors. These are question elements. Each questionelement is associated with a unique identifier. Each question element isgiven a RVU of 1. The assumption is that, the amount of time that ittakes to understand material to get the correct answer on one item isequal to the minimal time of relative value (T). An explanation of thecorrect answer may be provided for later use. Some embodiments providethat the stem can include images.

In some embodiments, question stems are associated with responses, whichare tagged as correct or as distractors. These are question elements.Each question element may be associated with a unique identifier.

Step 2. Question elements are tagged with one or more microcompetencycodes (microcompetencies). This can be accomplished in two ways. In someembodiments, each question element may include metatags where a code canbe associated. In some embodiments, question groups (QGroups) aregenerated and then question elements are placed under the appropriateQGroup.

Step 3. Question elements are sequenced into testing events. Eachtesting event may be associated with a summative analysis code. In mostcases this may be related to a course. A testing event may includequestion elements that are associated with one or more microcompetenciesand a RVU. Therefore, formative reports can be generated to associateindividual performance relative to a pass/fail threshold, relative tothe other individuals in a cohort, and/or by subject matter.

Using a learning management system, the cohort of individuals encountersthe testing event and data concerning the individual achievement on eachquestion element is recorded. A non-adjusted testing event report may begenerated.

Step 5. After all individuals, have completed the testing event, eachitem may be analyzed for quality of the item. The effect of the cohortmay be important at this juncture in the process. Every student mustencounter the summative evaluation so a post evaluation item analysiscan be performed on the raw results. In some embodiments, an institutionmay decide not to perform item analysis to generate a normalizedsummative report, but it is preferable to exclude poorly writtenquestions or questions where the cohort guessed.

It is noted that item analysis may include many statistics that canprovide useful information for improving the quality and accuracy ofmultiple-choice or true/false items (questions). Some of thesestatistics include item difficulty, which may be determined as thepercentage of students that correctly answered the item. This processcan be performed within the learning management system and/or throughoperations and methods disclosed herein. One function of the itemanalysis is to remove poorly constructed questions or questions wherethe entire cohort performed poorly. An institution can devise multiplemethods for this process. This disclosure does not provide the specificmechanism of item analysis, but it provides that this operation beperformed before a summative report is sent for analysis.

Step 6. Based on question element item analysis, individual questionelements may be eliminated from reporting. In some embodiments, itemsmay be deleted one item at a time, because a higher exam reliabilitycoefficient may be provided if a question element is deleted, and theitem-total statistics report is re-run to ensure we do not lower theoverall alpha of the exam.

Step 7. Following exclusion of flawed question elements, a didacticmodality summative report may be generated to give the individual theiradjusted score (% correct), the class average, the individual classrank, and/or an explanation of the items missed, among others. Theinstitution may choose to average these reports over courses to givetraditional grades. That process is not unique and is not in detailherein.

Alternative Step 7. As an alternative to the above-described operation,following the exclusion of flawed question elements, a didactic modalitysummative report may be generated to give the individual their adjustedscore (% correct), the class average, the individual class rank, and/oran explanation of the items missed, among others. Question elements maybe tagged with one or more microcompetency codes (microcompetencies).

Step 8. A didactic modality normalized summative report may be created.This data may be verified by the testing specialist as being completeand may be sent to a cumulative analysis engine (electronic circuit anddatabase(s)). In this manner, the RVUs associated with individualquestions may be replaced with an aggregation of RVUs by microcompetencyper individual for submission to the cumulative analysis grid.

Step 9. The didactic modality normalized summative report may beverified against the summative evaluation report to make sure that theindividual is receiving the same number of points relative to thecohort. This may be important to the process. If the overall goal of theprocess is to see where an individual student is strong or weak relativeto topics, different students in the same cohort can score the samepercentage of points, but have done well or poorly in differentmicrocompetency areas. This verification step assures that the samenumber of points are transferred during the normalization process.

Step 10. The verified normalized summative report may be submitted tothe cumulative analysis grid and it may be verified that the studentreceived the appropriate number of points to the appropriate competency.

Step 11. The verified normalized summative report may be archived as“committed” for audit purposes.

Experiential Modality

Still referring to FIGS. 1 and 2, experiential modality 40 may also beassociated with metadata codes 15 for outcome elements. The experientialmodality 40 can employ outcome elements that are formulated using(pre)tagged specific skills (e.g., “Individual Experience Element” or“IEE”). The individual experience elements (block 41) are electronicallyassociated with a respective microcompetencies 16, groups ofmicrocompetencies and/or sub-microcompetencies and RVUs 17 (block 42).Each student can encounter one or more individual experience elements(skills) 41 at different times and the number of events (N) (block 43)can vary from student to student. Proficiency in a skill provides thebinary decision (block 44) used by a post-event summative report 45 tosubmit the outcome element 48 for cumulative analysis 100 and/or anelectronic competencies assessment platform 100 p.

Placing a topic metatag (like microcompetency code) to a specific coursecomponent allows an institution to visualize where certain topics aretaught over the delivery schedule of the curriculum. From a practicalview, time units may be mapped in increments of 15 minutes, however, thedisclosure is not so limited. The didactic environment is the mostpredictable and is the closest to standardization. For example, 15minutes of lecture or presentation laboratory experience (cadaver lab,histology lab) is 15 minutes regardless of the subject matter.Therefore, it is substantially knowable and quantifiable for mostfaculty to agree upon the definition in order to report.

For experiential modality 40, microcompetency codes can be pre-definedwith respect to various actions, seminars, participation or viewingevents and procedures associated with an experiential environment of aparticular educational curriculum (e.g., clinical, surgical orlaboratory system for health sciences and practicals for observation orteaching in schools for a teacher curriculum). On a defined temporalbasis, e.g., daily, weekly or the like, an electronic report can begenerated (e.g., in a CSV format) which identifies student, RVUs,provider identification code and the respective microcompetencies. Thesereports can be generated daily and can accumulate over the academic life(and beyond) of each student. The data is provided with a convention forstudent identifiers (or a translator for allowing data input), and thesystem can be automated to create, evaluate and submit each report to agrid data repository and analysis circuit.

In health science, the performance in actual clinical procedures isimportant to properly evaluate individual performance. The individualsrecord these events in electronic patient record systems. Everyprocedure is tagged with an existing Procedure Anchor Code (PAC), whichis usually associated with the financial remuneration for the successfulperformance of the task, in medicine, these are the ICD-10 codes. Indentistry, these are the CDT codes. Examples of individual experienceelement evaluation steps are described below.

Step 1. Match each PAC with an appropriate microcompetency code. Allassessment reports will substitute the microcompetencies for the PAC.

Step 2. Each procedure that is represented by a PAC is evaluated forRVU. As noted above, the RVU measures the relative educational value foreach procedure. To normalize the outcomes assessment for experiential(clinical) and discussion-based educational environments, the RVU may bebased on three components. The first component is time, which may be thephysical time that it takes to perform a clinical task. In embodimentsin which the normalized value of one exam question is 15 minutes ofeducational investment, 15 minutes is equal to 1 RVU. The secondcomponent is laboratory time as many dental procedures involvelaboratory time for students. The amount of time that a student willperform laboratory tasks that are separate from clinical contact withthe patient may be estimated. The third component is higher expertise.For example, some procedures, like complex oral surgery, will involve ahigher level of interest by the student or a higher level of specialtyinstruction to perform in a pre-doctoral setting. Some embodimentsprovide that a multiplier of 3 may be used, although the multiplier maybe a value other than 3 in other embodiments. Some codes are“observational” in that students would not actually perform theprocedure and therefore only get credit for being involved.

In the experiential (e.g., dental clinical) examples that follow, a listof CDT codes was presented to a group of faculty members for theirestimate of an RVU for each code, “T” was previously defined as equaling15 minutes, so 15 minutes is equal to “1 unit”. The following formulawas used:RVU=(clinic time+lab time)×complexity multiplier.  (Equation 1)

For each institution, there is a core list of CDT codes that apply toall dentistry and there are certain procedures that are unique to thateducational environment. For example, every “house code” has an assignedRVU. Certain laboratory skills are taught in preclinical courses and canbe given PACs as derivative CDT codes. In this manner, the institutionmay use the clinical system to track laboratory outcomes. Examples ofclinical experiences with assigned RVUs and associated MC are:

-   -   PAC—D0421—Genetic test-oral diseases—36.00 Based on 2 hours of        clinical time (12 RVU), 1 hour of laboratory time, with HE        Multiplier. MC—01.02.09.01 Genetic Testing    -   PAC—D7287—Cytology sample collection—2.00 Based on 30 minutes of        clinical time. MC—01.08.01.08—Bacterial Cultivation    -   PAC—D1310—Nutritional counseling—8.00 Based on 1 hour of        clinical time, 1 hour of laboratory time.        MC—01.07.02.05—Nutritional Assessment

Step 3. Each day in the experiential curriculum, an individual mayperform procedures. A successful attempt may be given credit by an,appropriate authority. In some embodiments, all individuals in thecohort perform procedures as part of daily curriculum events. This iscalled a experience event report.

Step 4. An experience modality normalized summative report is generatedfrom the experiential platform to give the individual production ofprocedures and RVUs for each of the associated microcompetencies. Anexperience modality normalized summative report is created in a similarfashion to the didactic environment. This data is verified by theoutcomes specialist as being complete and is sent to a cumulativeanalysis engine. Some embodiments provide that the process replaces theRVUs associated with individual procedures with an aggregation of RVUsby microcompetency per individual for submission to the cumulativeanalysis grid.

Step 5. The experience modality normalized summative report is verifiedagainst the experience event report to make sure that the individual isreceiving the same number of points relative to the cohort. In thismanner, areas of an individual student's strength and/or weaknessesrelative to topics may be determined even if different students in thesame cohort can score the same percentage of points. The areas ofstrength and/or weakness may be identified by determining that a studenthas done well or poorly in different microcompetency areas. Thisverification step assures that the same number of points may betransferred during the normalization process.

Step 6. The verified experience modality normalized summative report issubmitted to the cumulative analysis grid and it may be verified thatthe student received the appropriate number of points to the appropriatecompetency.

Step 7. The verified experience modality normalized summative report isarchived as “committed” for audit purposes.

Discussion Modality

Still referring to FIGS. 1 and 2, the discussion modality 80 canelectronically tag student discussions (e.g., text or multi-media) postswith microcompetencies 16 and RVUs 17 after the student (or othertest-subject) encounter (block 84). In some embodiments, the discussionsubject can be associated with a defined (pre-tagged) microcompetencies16, but the RVU may be typically generated after the fact, based onstudent knowledge, responses and/or proficiency. The individualdiscussion elements (IDE) can be defined (block 81). For example, asdiscussed below in more detail in reference to FIG. 4, a discussionevent input screen may be provided for identifying, providing and/ordefining the individual discussion element. The IDES can be grouped intosummative discussions (block 82). Some embodiments provide that thediscussion modality 80 can be an interactive electronic (e.g., online)environment forum that a student or other test subject can respond to agiven problem, question or other prompt.

In complex educational environments, the ability to solve problems frompractical discussion of cases or problems may be difficult to evaluateand track. Accreditation bodies may place a great deal of value on theability to apply knowledge. Since the discussion itself can crossmultiple topics and can involve different levels of complexity, thediscussion events may be typically tagged for educational value separatefrom the event itself.

While embodiments disclosed herein contemplate that, text based postingswill be a viable means of providing a discussion forum, it is alsocontemplated that online multimedia communications may also be used fora discussion modality 80. Combinations of these types of discussionformats can also be used. Some embodiments provide that video streams ofthe multi-media video may be electronically stored with a summary ofevaluation for cumulative analysis. Services related to onlinemultimedia communications may be provided by a third-party onlinemultimedia communications service provider, which may be, e.g., aconsumer videoconferencing service provider such as Skype, MicrosoftLive Messenger, Yahoo! Messenger, America Online Instant Messenger,and/or Apple iChat, among others.

In some embodiments, the discussion modality 80 can be carried out usingand/or including a threaded discussion logged by student with timeposting. The discussion posts can augment basic blog technology with aRSS (Really Simple Syndication) client. RSS allows for subscription,management and posting of content to secure blog systems. In thismanner, the user may make postings to the blog without launching abrowser. Current RSS clients are useful models for binary applicationsin order to give rise to properly engineered applications specificallyengineered to meet the complex needs of case-based education.

However, computer applications for writing, managing, and, participatingin cases can be written that may be more suitable for largerschools/practitioner implementation. Using the case application suite,an implementation (on-boarding into a central system or use in discretestandalone systems) with multiple schools, practices, and programs canbe facilitated.

Some embodiments of the invention seek to provide participating,educators with an implementation strategy for case-based education thatcan actually be scaled to fulfill the educational mission to teachcritical thinking and problem solving. From an educational philosophystandpoint, educators may disagree concerning the number of cases, thedepth of cases, the role of the instructor, and the outcomes assessmentof individual implementations. From a technology standpoint, the systemscan be powerful enough to facilitate the educational mission whilesimple enough to encourage use.

For a discussion modality 80, it may be desired to include cases thatprovide fact patterns that are authentic, promote realism and yieldintense learning experiences that the practitioners and/or educators canrelate to students. Beyond recruiting “non-traditional” cases, thetechnology for writing the ease components, attaching related content,and creating learning objectives may be consistent. Faculty resourcesmay limit the time within the schools to reformat each practitionercase, and the alternative is to limit the number of practitionersubmissions. For some disciplines (e.g., dental and medical), toadequately assess competency, it is believed that there should behundreds, if not thousands, of discussion cases available to students.The preferred case writing application should provide simple processingtools for creating the components, for reediting components, and thenshould package the resultant case so the components cannot be altered.

Managing cases, may be a different experience from writing. Each schoolcan have a different role for cases. Each school can have differenttheories for student and faculty grouping. Each school can also havediffering views for outcomes measurement. In some embodiments, thesystems, and methods disclosed herein can be configured to accept a casepackage from a case-writing tool and allow the course director to assignstudents and faculty, to determine posting times and resolution dates,and/or to design appropriate grading criteria, among others. As apractical matter, this application environment would adapt individualcases to meet larger curriculum goals. Participating in cases should berelatively simple. Once the management application assigns a case to astudent or faculty, the participation tool for the discussionenvironment should: alert the user to the assignment; “push” thepostings to the client through simple subscription; allow for directposting; and monitor time components and grading issues.

It is believed that there will be many users of the participation tool,fewer users of a case writing tool, and very few users of the casemanagement tool. Practitioners may propose or submit cases that otherpractitioners could take for CE credit. Students may write cases forother students. Issues that currently restrict school and program use ofcases, such as number of cases, and number of faculty, could be reduced,if not eliminated.

In some embodiments, evaluating performance corresponding to thediscussion modality 80 may include exemplary operations as provided inthe following steps.

Step 1. A discussion group is created with one or more individuals. Thisgroup will all receive the same credit as each individual. Theindividuals participate in a collective. The cohort can be, and usuallyis divided into sub-cohorts to facilitate discussion. In the currentimplementation, the typical sub-cohort has 5-10 students. The discussionsub-cohort summative report can be generated (block 83). The individualdiscussion element 81 can be meta-tagged with metadata codes includingmicrocompetencies 16 and RVUs 17 (block 84). For example, typically, atleast the RVU is defined and tagged (subjective with guidelines) by agrader. The microcompetencies may also be applied at that time, but mayalso be generated earlier based on defined topics rather than “stream ofthought” type discussion. The definition of outcomes cohort can begenerated (block 85) as well as a post-element analysis summative report(block 86). The outcome elements for summative evaluations 88 can besubmitted to the cumulative outcome storage data collection 100.

Step 2. A discussion may be initiated with a question or prompt. Withinthat thread, individuals may respond to the prompt and to theparticipation of others in the group.

Step 3. An individual discussion event may include a unique itemidentifier, a unique thread identifier, a time stamp of the posting(including date and time), a unique user identifier, and/or the body ofthe posting, among others.

Step 4. A discussion may be limited by time. Based on the time stamps ofthe discussions all of the IDEs within a proper reporting interval, adiscussion sub-cohort summative report may be generated for assessment.The discussion sub-cohort summative report may be verified and sent forformatting by the discussion evaluation tool. This process may presentthe discussion for third party evaluation.

The discussion sub-cohort summative report may be submitted for“grading” which will attach corresponding microcompetencies and RVUs toeach post. An evaluator, grader, host, other prompt and/or otherstudents can interact with the test student(s) to assess depth ofknowledge, problem solving skills and the like. The RVU may be partiallybased on subjective criteria and partially based on objective criteria(e.g., keywords, length of text, discussion time, and the like). Thesystem can accept a post-discussion summative report that attaches asummary of grading of the discussion with the discussion text itself forfuture retrieval. The subjective weighting may be provided by theinteractive person “grading” the student/test taker, or groups ofstudents, and is typically within a predefined range of based on timeincrements of 15 minutes and difficulty. In some embodiments, a simplepost may be worth at least 0.1 RVUs.

Brief reference is now made to FIG. 3, which is a screen shot 80 s of auser interface in which a post is being graded. The screen shot 80 sincludes a table-format summary of: Post Author by name (or studentidentifier), a Type of input (e.g., Student Post Content, Student PostLogistics, Student Post Other, or if Faculty Advisor, Faculty Post Case,Faculty Guidance or Faculty Other), microcompetencies corresponding toeach post content and logistics, etc. with associated RVUs and Comments.A Student summary window 80 w may include a summary of numbers of postsand total RVUs for each student and overall for the IDE. An evaluator“submit” input may be used to submit the data to an evaluation, circuitonce the IDE is complete with RVUs and microcompetencies.

Brief reference is now made to FIG. 5, which is an example of a CSV filefrom a Yammer® discussion. The evaluation and tagging of discussioncontent can be facilitated by a dynamic survey. It is contemplated thata report (e.g., generated from Yammer®, for example) can be used tocreate a dynamic “survey” using a defined survey tool, for a facultymember or other defined person to grade the discussion events. As notedabove, the report can provide a word count for the body of the post.

Reference is now made to FIG. 4, which illustrates a screen shot 80 m ofa user interface that may be used for the discussion modality 80 toallow an evaluator to electronically assign microcompetencies andassociated RVUs for an IDE 81 (FIG. 1) for a student and posting type. Aword count may be generated and displayed. For example, as illustratedthe word count is 75/100. A progress to completion of topic (potentiallywith a time remaining reminder) input and a comment input section may beincluded. User inputs such as “Next” and/or “Submit” may be provided forthe evaluator or other user to proceed to a next step or to submit thedata. An “Overview” user input may provided for a user to toggle to anOverview screen. After all posts have been graded, a discussionsub-cohort normalized summative report may be generated. All of themicrocompetency codes may be verified as valid, but there is no raw datato verify against.

The sub-cohort information may be very useful at this juncture. Eachsub-cohort of the cohort provides multiple opportunities for RVU pointassignment. Unlike the didactic environment, each sub-cohort has uniquediscussions and posts. The decision may be made by the outcomesspecialist to give each individual their own grade based on theirpersonal posts, and/or to give all sub-cohort participants credit foreveryone's participation. This is a choice that may be made based on thediscussion environment and the curriculum needs. At the end of all ofthe posts (text or multi-media), there can be a place for a “groupgrade” of pass/no pass. There can also be a place for an individualgrade of passing pass beside the name of the student/user.

The verified discussion sub-cohort normalized summative report may besubmitted to the cumulative analysis grid and verified that the studentreceived the appropriate number of points to the appropriate competency.The verified discussion sub-cohort normalized summative report may thenbe archived as “committed” for audit purposes.

For each post, the grader can evaluate one more of the following:

-   -   (1) Posting Type: From a pull-down there are a number of        possibilities, shown below as six possible choices:    -   “Faculty Post Case”    -   “Faculty Post Guidance”    -   “Faculty Post Other”    -   “Student Post Content”    -   “Student Post Logistics”    -   “Student Post Other”    -   (2) Microcompetency Code(s): The input can include a plurality        of fields, e.g., 3 fields, where microcompetencies associated        with the post can be entered (a user must then elect the 3        closest microcompetencies, the system may provide a keyword        search of the post and suggest microcompetencies that may be        appropriate).    -   (3) Relative Value Units: This input is typically limited by a        range of 0-10, such as, for example, a field limitation which        may be implemented or selected by a user via a Pulldown with        numbers, e.g., 1 to 5.    -   (4) Comment: A field that assumes no comment, but where a        message/paragraph can be entered.        Cumulative Outcomes Storage

In some embodiments, the system 10 can be configured so that commonnaming and coding of students is used in all modalities and/orenvironments and/or that appropriate translators are used to importand/or exchange data between the various systems and/or the cumulativeanalysis engine.

Where all three environments are used (modalities 20, 40, 80), allverified normalized summative reports (with outcome elements) from allmodalities 28, 48, 88 can be submitted to at least one data repository90 (e.g., archived student education history server). Typically, thereports (e.g., outcome elements) may be provided as they are generatedor completed, but may also be provided on a time-based input (upload orother data transfer).

Each of the outcome elements that is stored in the common datarepository may include the following fields:

Unique Element ID

Unique Program ID

Unique Student ID

Date

Didactic, Clinical, Discussion

Primary, Remediation

Microcompetency

RVU

The fields do not have to be in a specific order, provided that thesource file can map to these elemental fields.

Analysis Grids

A cumulative analysis module 100 (FIGS. 1, 2) can be configured toanalyze data from one, two or all the environments corresponding tomodalities 20, 40, 80 for a respective student over time or at, aparticular desired time. Thus, as shown in FIG. 6, the systems/methodscan generate a cumulative evaluation grid 200 also known as a cumulativeanalysis or competencies grid. The cumulative analysis module 100 can behoused in one server or host or may be distributed. Additionally, thecumulative analysis module 100 and/or the data repository 90 may beprovided using distributed computing resources, such as, for example,cloud-based data storage and/or processing.

Data cross-section is a basic expression of difficult data. The datafrom educational outcomes may be specifically difficult to express.Advantages from methods and systems disclosed herein may be realizedbased on the concept of competence itself. A “competency” or “competencystatement” is a synthetic aggregation of related skills or topics.Competencies are extensible by definition. Any program defines theseextensible concepts based on their own concepts and approaches. The grid200 is the expression of the data from all of the sources in methodsthat show individual student data relative to all students in the cohortand relative to the relevant topics that represent competence for thecohort.

Education may be difficult and complex. Every student enters a neweducational experience with previous knowledge and different abilities.Every program within a discipline tries to provide experiences that areengineered to train a student to become capable of being a member of aspecific workforce. That could be a chemist, an author, a dentist, anengineer, or any of another myriad specialties.

Educational programs present the students with a series of experiences,called curriculum, and evaluate performance with many different metrics.Systems and methods disclosed herein create a way to “normalize” thevarious outputs of curriculum evaluation to simplify the visualpresentation of this data.

In the previous discussion, the systems/methods for the creation of thegrid data expression is addressed. In this section, the data ismanipulated to better graphically represent the results for educationaldecision-making.

At the cross-section of the student and the competency is the “gridcell”. Each grid cell 201 is unique to that grid and that cohort.Depending on the data allowed, the grid cell 201 calculates thatstudents' performance for the respective microcompetencies defined forthat competency. The sum of all of the points and partial points arerepresented in one number that may be expressed to, for example, thetenths decimal place. Two separate events may be calculated based on theindividual grid cell 201. First, the total student performance may becalculated for each student in the cohort. Second, the student data fora specific competency may be analyzed for various rankings. The detailsfor each grid cell 201 can be attained currently with a combination ofspecific keys.

A grid cell 201 that detects no data for display may be represented with0 and with specifically colored background to denote a lack of data. Forexample, some embodiments provide that a grey background may, denote alack of data. Showing no data within a grid cell 201 is not unusual inthe early parts of a curriculum, however a hole in the latter stages oftraining may show a curricular deficiency.

In some embodiments, the sum of all grid cells 201 may be summed in anumber to a defined degree of accuracy, e.g., as shown to the tenthsdecimal place in a separate column. The rows may be auto-sorted fromgreatest number to lowest number with a result of ranking the studentswithin the cohort. With the addition of new content the rows mayauto-sort and result in new rankings.

The grid cells 201 may then be evaluated vertically for each competency.Some embodiments provide that the data from all of the grid cells 201may be mathematically sorted into three to seven standard deviations.Some embodiments provide that the data is sorted into five standarddeviations. The highest standard deviation values may be representedwith a gold background to the related grid cell 201. The lowest standarddeviation may be represented with a red background to the related gridcell 201. The second, third and forth standard deviations may berepresented by shades of green, for example, from lightest to darkest,respectively. The result is a graphic display that allows theadministrator to see how students rank based on the specific topicswithin a grid 200. Additionally, the areas of specific weakness for astudent “lights up” in red and areas of specific strength are shown ingold. This allows the program to target remediation of specific weaknessand recognition of specific strengths. The colors disclosed herein areby way on non-limiting example in that other colors may be used withinthe scope and spirit of the present invention.

Two additional features can alter the standard deviation color-coding.These may be referred to as a “hard floor” and a “glass ceiling”. Theadministrator can place number values in the grid 200 to represent aminimal value that is required to be competent. This “hard floor” willset, a value below which the number will be represented, as red,regardless of the standard deviations. This may serve to set minimalstandards for numbers of procedures that must be accomplished. The glassceiling manual designation is a number above, which all grid cells 201will be designated as gold. This “glass ceiling” allows theadministrator to determine a threshold that represents excellence,regardless of the standard deviations. In this manner, students can goabove this number and the entire cohort can gain this level ofexcellence.

In some embodiments, the high (gold) cutoff and low (red) cutoff maydefault to the standard deviations unless specifically entered by theadministrator. The grid 200 may automatically calculate a Student High,a Student Low and an Average for each Microcompetency column.

The grid 200 can be of a single topic with multiple associatedmicrocompetencies or based on other topics or classifiers of interest.Each cell 201 of the grid represents an intersection of the student andtheir performance (RVUs) filtered by the specifically includedmicrocompetencies and by the included environments. No value can berepresented in a grid cell 201 as equal to zero RVUs or can be leftblank. For statistical purposes a blank value may be equal to zero RVUs.Some embodiments provide that the rows of cells may represent respectivestudents in the cohort and the columns of cells may represent therespective microcompetencies, however, such arrangement is non-limiting.

The grid 200 may change with the frequency that inputs are provided. Forexample, some embodiments provide that the grid 200 may automaticallyupdate daily if inputs are provided daily. For example, daily reportsfrom experiential environments will provide points associated withmicrocompetencies that will accumulate over time to the grid. The samemicrocompetencies can show up in multiple areas. Similarly, for didacticinputs, exam reports may provide normalized points from the didacticexams and can be provided for cumulative analysis. For both the didacticand experiential inputs, some embodiments provide that an AXIUM® projectmanagement software may be used, Axium XTS, Inc. Oregon, USA, it isbelieved that AXIUM® has an extensible metatag called “category” thatcan be adapted for the microcompetency code to avoid the use of atranslator, which may reduce potential implementation errors.

The grid 200 can be interactive as noted above. The grid 200 canpresented on a display with a UI (User Interface) such as a GUI (GraphicUser interface) that allows a user to select a student to reveal moredata associated with the student, to select microcompetencies toelectronically automatically reveal various sub-topics and associatedscores. In this manner, a user can analyze trends with the student data,e.g., search for common factors for students failing, for students inhonors ranges and the like. Thus, for example, if a number of morestudents that are identified as failing are in the same class, perhapsthat is an indication that there is a problem with the class.

The interactive grid 200 can be configured to allow users to click anddrag the table to navigate and ctrl-click, select and/or touch (contactor touch gesture) a cell 201 for cell-specific information. For example,student ID numbers and competency score cells can be clicked to showdata points used to create the selected cell's content.

A user can drill down to show groups of students for differentcriterion, i.e. year in program, gender, and the like. The grid 200 caninclude cell information pop-up comments and the information accessibleand/or shown when a user select (e.g., ctrl-clicks) a cell can bedefined by the type of user accessing the grid.

The cumulative data in the evaluation may identify other common factorsto allow for pro-active adjustments in the curriculum, educationalresources and/or for the student.

In some embodiments of the grid 200, the didactic environment summativegrading inputs may have a much smaller weighting of relevance in thecumulative evaluation than either of the experiential or discussionenvironment grading inputs. For example, about 10% of an overallcognitive assessment score for a particular microcompetencies can bebased on didactic summative grading, compared to about 40-60% forexperiential and 30-50% for discussion summative grading.

The grid 200 allows extensible definition, of “competencies” as subsetsof microcompetencies. As data accumulates to the data repository, thegrid can dynamically calculate performance from all three environments.As noted above, minimal thresholds and performance rewards can betagged.

As briefly discussed above, the elements of mapping using common topiclogic, called microcompetencies, may provide a hierarchical numberinglabel for topics. In some embodiments, there are 4 levels of thishierarchy separated by legal numbering periods. Examples are as follows:

-   -   02—Designates Body System    -   02.08—Designates Gastrointestinal System    -   02.08.07—Designates Clinical Dentistry    -   02.08.07.13—Designates Resin Restorations    -   01—Designates Pan-systemic Disciplines    -   01.06—Designates Human Immunology    -   01.06.08—Designates Immunizations    -   01.06.08.01—Designates Vaccines

Some embodiments provide that not all topics, have 4-level detail. Forexample, some outcomes may adequately test the 3rd level as a group. Inthe numbering scheme, a 00 may be added in the fourth level of the code.

Different educators within a curricular program may have differentstudent evaluation needs, therefore different views of the total body ofoutcomes are needed. A “competency grid” may define related topics forsimultaneous viewing. For instance, there may be a need to see how thestudents perform in human anatomy. A grid can be created to representall of the anatomic microcompetencies. The data may be parsed by anatomyby system. In this example, anatomy is the basis of the grid, eachsystem represents an extensible competency, and the microcompetenciesfor the anatomy of that system defines the student performance that willbe represented in the grid cell 201 of the grid 200.

For the purpose of the grid 200, the definition of each competency maybe a simple list of the codes that the administrator considers to definewhat needs to be displayed. An example for the anatomy of thecardiovascular system follows:

-   -   CVAS—Normal Development and Structures    -   0.2.06.01.00    -   02.06.01.01    -   02.06.01.02    -   02.06.01.03    -   02.06.02.00    -   02.06.02.01    -   02.06.02.02    -   02.06.02.03    -   02.06.02.04    -   02.06.03.00    -   02.06.03.01    -   02.06.03.02    -   02.06.03.03    -   02.06.03.04    -   02.06.03.05    -   02.06.03.06    -   02.06.03.07    -   02.06.03.08    -   02.06.03.09    -   02.06.03.10    -   02.06.03.11    -   02.06.03.12    -   02.06.03.13    -   02.06.03.14    -   02.06.03.15

Once the codes, are defined, the second factor that filters theexpression within a grid cell 201 is the data source. As describedpreviously, there is data from the discussion environment, didacticenvironment and the clinical environment. Any grid 200 can display thedata in a grid cell 201 from the designated microcompetencies from anyone or combination of data sources. A grid can be made to show all dataor just the data from didactic exams.

In practice, some students are better on exams than they are in clinic.This will show in the expression of filtered grids 200. As describedabove, different administrators and educational stakeholders may needdifferent reports from the curriculum. The creation of specific gridsallows these customized views.

Similar to data source filtering is the inclusion of remediationelements. Each data source element may also be designated as primary orremediation. Primary data may represent outcome elements that everystudent experiences. Remediation data may represent outcome elementsthat are targeted to re-test certain students for specific deficiencies.In this manner, the administrator may create grids that only use primaryfor an evaluation of the entire cohort. This allows specific grids to bemade to show additional work that is given to certain students. Thisprovides that the grids that would allow for the expression ofremediation can include “hard floor” designations in the grid to showwhen a student has achieved a defined level of competence.

Certain grids are made to give a very broad analysis of the completecurriculum. In practice, this can result in a multiplication error. Forexample, if the administrator is not careful, the same microcompetencycan be represented in multiple competencies in the same grid. The bestpractical example is a grid that has both systems wad disciplinesrepresented. If the same microcompetency is in multiple competencies,and that microcompetency has several outcomes, there will be amultiplication effect error that incorrectly affects the studentrankings. The achievement or deficiency in a specific microcompetencywill be compounded relative to single microcompetencies. In someembodiments, that may be desired, but the skewing of the data expressionmay be anticipated.

Brief reference is made to FIGS. 7A and 7B, which illustrate twohierarchical competency lists (that can be used for a competency grid)with an exemplary list of associated microcompetency codes. The 00.01 etseq. list is for “Quantitative Methods” while the 00.02 et seq. list isfor “Basic Genetics and Embryogenesis”. In this manner, an extensiblemechanism for evaluating the quality of a curriculum that is responsiblefor defining competency may be provided. Additionally, all three typesof educational techniques can be evaluated concurrently. Thenormalization processes allow every adopter to customize their analysis,as desired. Further, each institution may customize the evaluation toole.g., instead of 15 minute time normalization, shorter or longerstandards may be used, e.g., 5 minutes or 30 minutes. Each institutionmay then generate different microcompetencies that may be furthernormalized when comparing between institutions.

Brief reference is now made to FIG. 8, which is a schematic illustrationof a dashboard 300 that restricts the type of data that can shown todifferent users. Some embodiments of the system 10 can restrictinformation/functionality based on who logs into the dashboard 300.Initially, the users may be characterized as one of three types of usersthat will be accessing the dashboard. The dashboard (or portal) candefine more specific access rights as additional dashboard elements arecreated (Yammer® grading, and microcompetency management). Examples ofuser types according to some embodiments include:

User: (Teacher, Professor 301)

-   -   access all information and functionality

Educational (University) Admin 302

-   -   Can only view competency grid. (no access to microcompetency        management or yammer areas    -   FERPA: to comply with FERPA the Student ID column can be omitted        as needed.

Student 303

-   -   Can only view their scores in the competency grid 200. The        competency grid 200 may be customized to show only the student        data, but the grid 200 may also show student high and class        averages.        Other users 304 may include teachings assistants, staff,        advisors, publishers of educational resources (identify        materials that provide better student outcomes or those that        need improvement), teacher evaluation functions (for awards or        correctional help), accreditation services, and/or licensing        boards, among others. Again, as needed to comply with privacy        rights, employment laws and the like, the type of data presented        to different users can be controlled.

Some embodiments disclosed herein may be particularly suitable forevaluating health-science students, schools, classes, educationmaterials (e.g., books) and/or curriculums. However, other embodimentscan be used to evaluate other students, schools, curriculums, teachers,classes, resource books and the like. The term “health-science” refersto medical related educational fields, including nursing, dental,pharmacy, medical doctors, veterinarians, psychiatrists, psychologists,physical therapists, other therapists and practitioners, particularlythose health/science fields where board certification may be requiredfor practice in a particular field.

Many educational programs may require proof of knowledge, skills, andinter-disciplinary problem solving. Some embodiments provide systems andprocesses for a continuous (over time) and, optionally, substantiallysimultaneous analysis of performance from didactically-focused,skill-based, and problem-based environments. Educational outcomes can beforecasted and cognitive success identified in a pro-active manner.

Referring to FIG. 2, the system 10 can include an electroniccompetencies assessment platform 100 p. The term “competenciesassessment platform” refers to a module, circuit, and/or processor thatcan accept data from and/or integrate a combination of different systemsand defined variables for analysis of cognitive outcomes incompetency-based education environments related to a defined set or setsof microcompetencies related to one or more competency areas. Thecompetencies assessment platform 100 p may creates an ongoingsubstantially constant (e.g., updated over time, but not necessarilyreal time) cumulative analysis of competencies as defined by formativeand summative evaluation components.

Some embodiments disclosed herein contemplate that substantially alloutcomes from all platforms build toward competency. Thus, all formativereports can be aggregated into a common analysis if all events use thesame criteria. For example, each individual can be given credit for acertain number of points per microcompetency per event. If the cognitiveoutcomes for a competency can be represented as an aggregation ofmicrocompetencies, then an infinite number of cumulative analyses can begenerated from the same data set. Examples of some steps that can beused for cumulative analysis steps are summarized below:

Step 1. The cumulative data storage is defined as one or more datarepository for every outcome event for an individual. Regardless of thesource platform (didactic, clinical, or discussion), the individual maybe given credit for points associated for each microcompetency.Thousands of these events may accumulate over time, platform, andassessment.

Step 2. Cumulative analysis mapping as disclosed herein may provide aninstitution with the ability to define how the raw data will beaggregated for display and analysis. A competency is a statement of thesubset of content that an institution uses to evaluate performance. Themapping allows the institution to define a competency in terms ofdifferent combinations of microcompetencies. Some embodiments providethat there can be multiple maps of differing detail. Each competency mayaggregate an individual's performance by combining the performance ofeach microcompetency. The assumed cross-reference is a set ofindividuals that make up a cohort. The mapping allows the institution todetermine what individuals make up a cohort.

Step 3. Additional cross-cohort data can be assessed and placed in thesame grid. For example, maximum performance, minimum performance, cohortaverages, minimal achievable levels, and other items of interest may beassessed and placed in the grid.

Step 4. Desired data to be shown in a display grid 200 or grids can bedefined.

Some embodiments of the invention may use a computing architecture inwhich the user interface, the application processing logic, and/or theunderlying database(s) can be encapsulated it logically-separateprocesses. In any given application utilizing this type of computingarchitecture, the number of tiers may vary depending on the requirementsof the particular application; thus, such applications are generallydescribed as employing an n-tier architecture. See, e.g., Exforsys.com,N-Tier Client-Server Architecture. For instance, some embodiments of theinvention may employ a 2-tier architecture, commonly referred to as aclient-server architecture, wherein a client application such as a webbrowser makes a request from a web server, which processes the requestand returns the desired response (in this case, web pages). Otherembodiments of the invention may be structured as a peer to peer or a3-tier or other larger multi-tier architecture. For the latter, the webserver provides the user interface by generating web pages requested bya web browser, which receives and displays code in a recognized languagesuch as dynamic HTML (Hypertext Markup Language); middleware executingon an application server handles the business logic; and databaseservers manage data functions. Often, the business logic tier may berefined into further separate tiers to enhance manageability,scalability, and/or security.

Accordingly, in some web-based hearings services, the web applicationscan use a 3-tier architecture with a presentation tier, a business logictier, and a student record data tier. The web application tiers may beimplemented on a single application server, or may be distributed over aplurality of application servers. For example, the presentation tier canprovide the discussion modality 80 using web pages that allow a user torequest student responses and allow communication between the studentand an educator (e.g., teacher or professor). The presentation tier maycommunicate with other tiers in the application such as the businesslogic tier and/or student record data tier by accessing availablecomponents or web services provided by one or more of the otherapplication tiers or by third party service providers. The presentationtier may communicate with another tier to allow authorized users toaccess student record data and/or database stored microcompetency codes,procedures, instructions, or protocols. The business logic tier cancoordinate the application's functionality by processing commands,restricting user access and evaluating data. The functionality of thebusiness logic tier may be made accessible to other application tiersby, for example, the use of web services. The business logic tier mayalso provide the logic, instructions or security that can separate anddistinguish users. While the student data record tier can hold theprivate student records data and encapsulate, such records fromunapproved parties so as to comply with FERPA or other privacyregulations. The student records data tier can make data availablethrough, for example, stored procedures, logic, instructions and thelike accessible, for example, by web services.

FIG. 9 is an example of method steps that can be carried out accordingto embodiments of the present invention to evaluate students cognitiveprogression in a competency-based manner. As shown, RVUs for didactictesting events, correlated to student and microcompetencies, aretransmitted to and collected by a student data record repository (e.g.,database with memory such as one or more servers) (blocks 208, 225).Similarly, RVUs for experiential events, correlated to students andmicrocompetencies are transmitted to and collected by a student datarecord repository (blocks 210, 225). RVUs for discussion events,correlated to students and microcompetencies are transmitted to andcollected by a student data record repository (blocks 220, 225). Thesame or different repositories may be used, e.g., different ones fordifferent students, schools, different type of student record, and thelike. Cumulative analysis reports can be generated for respectivestudents by summing collected RVUs by microcompetencies (block 230). Thereports can be generated automatically over time daily, weekly, monthly)and/or upon request. The minimum threshold that establishes satisfactorycognition for a particular topic (e.g., microcompetencies) can bechanged over time by a defined user (not student) to account foreducational progression. The reports can be customized to block data orpresent only defined fields of data, depending on user-based accessprivileges as discussed above. The reports (particularly, where studentidentifiers are present) can be sent to an email account or placed on asecure (restricted) web portal. The student can define how often toreceive such a report at log-in or set-up (or such a report may be basedon a default action), or a student may request a report by accessing theweb portal. Some systems may automatically send the student a reportwhen a cumulative summary report indicates that one or moremicrocompetencies scores is below a desired threshold at that point intime.

As will be appreciated, by one of skill in the art, embodiments of theinvention may be embodied as a method, system, data processing system,or computer program product. Accordingly, the present invention may takethe form of an entirely software embodiment or an embodiment combiningsoftware and hardware aspects, all generally referred to herein as a“circuit” or “module.” Furthermore, the present invention may take theform of a computer program product on a computer usable storage mediumhaving computer usable program code embodied in the medium. Any suitablecomputer readable medium may be utilized including hard disks, CD ROMs,optical storage devices, a transmission media such as those supportingthe Internet or an intranet, or magnetic or other electronic storagedevices.

Computer program code for carrying out operations of the presentinvention may be written in an object oriented programming language suchas Java, Smalltalk C# or C++. However, the computer program code forcarrying out operations of the present invention may also be written inconventional procedural programming languages, such as the “C”programming language or in a visually oriented programming environment,such as Visual Basic.

Certain of the program code may execute entirely on one or more of auser's computer, partly on the user's computer, as a stand alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer. In the latter scenario, theremote computer may be connected to the user's computer through a localarea network (LAN) or a wide area network (WAN), or the connection maybe made to an external computer (for example, through the Internet usingan Internet Service Provider). Typically, some program code executes onat least one web (hub) server and some may execute on at least one webclient and with communication between the server(s) and clients usingthe Internet.

The invention is described in part below with reference to flowchartillustrations and/or block diagrams of methods, systems, computerprogram products and data and/or system architecture structuresaccording to embodiments of the invention. It will be understood thateach block of the illustrations, and/or combinations of blocks, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general-purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the block or blocks.

These computer program instructions may also be stored in a computerreadable memory or storage that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer readablememory or storage produce an article of manufacture includinginstruction means which implement the function/act specified in theblock or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing the functions/acts specified inthe block or blocks. The Internet can be accessed via any desired devicehaving access to the Internet including wireless or hard-wiredcommunication systems (such as cellular telephones), PDAs, desktop orportable computers including lap or handheld computers, notebookcomputers, and the like.

Referring to FIG. 10, in some embodiments, the system 10 includes atleast one web server 310 (which may be provided by an onlinecommunications provider such as Yammer®) and a plurality of web clients3351-3352. Although illustrated as two web client, the number of webclient may be substantially more than two and may vary by institution(numbers of participating students administrators andteachers/professors or other educators), typically, is between100-10,000, for a respective institution, or even more, corresponding tothe number of registered users. Some of the users can communicate withthe system 10 via any suitable device having website browsingcapability, including, for example, PDAs and/or cellular telephones 3353as shown in FIG. 10. Thus, for example, a professor user can communicatewith the student user during a discussion event via the Internet 300using a FDA (personal digital assistant), notebook or cellular telephonehaving web-browsing capability (or palm, laptop or desktop computer).

The at least one web server 310 can include a single web server as acontrol node (hub) or may include a plurality of servers (not shown)providing a web portal 310 p. The system 10 can also include routers(not shown). For example, a router can coordinate privacy rules on dataexchange or access. Where more than one server is used, differentservers (and/or routers) may execute different tasks or may share tasksor portions of tasks. For example, the system 10 can include one orcombinations of more than one of the following: a security managementserver, a registered participant/user directory server, a student recordmanagement server, and the like. The system 10 can include firewalls andother secure connection and communication protocols. For Internet basedapplications, the server 310 and/or at least some of the associated webclients 35 can be configured to operate using SSL (Secure Sockets Layer)and a high level of encryption. Furthermore, given the ubiquitous natureof the Internet, web-access devices may readily be moved from site tosite. Additionally, additional security functionality may also beprovided. For example, incorporation of a communication protocol stackat the client and the server supporting SSL communications or VirtualPrivate Network (VPN) technology such as Internet Protocol SecurityArchitecture (IPSec) may provide for secure communications between thestudent sites and other sites to thereby assure privacy.

The server 310 can provide a centralized administration and managementapplication. The server 310 can be configured to provide sessionmanagement, tracing and logging systems management, workload managementand member services. The server 310 can include or communicate with aplurality of databases including participant/user profiles, a securitydirectory, routing security rules, and student records. The server 310can include several sub-servers for integration into web systems, suchas, but not limited to, a web application server (WAS) which maycomprise an IBM WebSphere Application Server, a Directory Server such asLDAP directory server, and may include an Anonymous Global PatientIdentifier (AGPI) Server, a DB2 Server, and a Simple Mail TransferProtocol (SMTP) Server. It is noted that although described herein as“servers” other suitable computer configurations may be used. The server310 can be configured with web application functions that appear atportal sites. The server 310 may comprise and/or be configured as a WebSphere Business Integration (WBI) server. The web server 310 can includea web-based administration application. The web application can be usedto: allow a user to register as a participant, manage Access ControlLists (ACLS), logon using universal ID or password access, logoff,define profile preferences, search, participate in discussion events andthe like.

The web clients 3351-3352 can be associated with different users anddifferent user categories or types. Each category or type may have adifferent “privilege” or access level to actions or data associated withthe systems 10. For example, the systems 10 can include student users,administrative users, and teacher/professor users, each of which canhave, different access levels or restrictions to data and/or actionsallowed by the system.

The web clients 3351, 3352 can be distributed at different geographiclocations in different time zones and states or even countries. In otherembodiments, the web clients 35 can be at a single educational center.Different user types may be at different geographic locations.

As noted above, the clients may include webcams or cameras to allow formultimedia communication during some discussion or some experientialevents, for example.

FIG. 11 illustrates an exemplary data processing system or databaseenvironment that may be included in devices operating in accordance withsome embodiments of the present invention. As illustrated in FIG. 11, adata processing system 116 which can be used to carry out or directoperations of the hub and/or web application (e.g., comprising anAdministrative Server) includes a processor 138, memory 136 andinput/output circuits 146. The data processing system may beincorporated in, for example, one or more of a personal computer,server, router, or other device with web access/functionality. Theprocessor 138 communicates with the memory 136 via an address/data bus148 and communicates with the input/output circuits 146 via anaddress/data bus 149. The input/output circuits 146 can be used totransfer information between the memory (memory and/or storage media)136 and another computer system or a network using, for example, anInternet protocol (IP) connection. These components may be conventionalcomponents such as those used in many conventional data processingsystems, which may be configured to operate as described herein.

In particular, the processor 138 can be commercially available or custommicroprocessor, microcontroller, digital signal processor or the like.The memory 136 may include any memory devices and/or storage mediacontaining the software and data used to implement the functionalitycircuits or modules used in accordance with embodiments of the presentinvention. The memory 136 can include, but is not limited to, thefollowing types of devices: cache, ROM, PROM, EPROM, EEPROM, flashmemory, SRAM, DRAM and magnetic disk. In some embodiments of the presentinvention, the memory 136 may be a content addressable memory (CAM).

As further illustrated in FIG. 11 the memory (and/or storage media) 136may include several categories of software and data used in the dataprocessing system: an operating system 152, application programs 154,input/output device drivers 158, and data 156. The application programscan include a User Registry Module 120, a Microcompetency CumulativeAnalysis Module 124, a Student Data Records Module 125, and the like.The data 156 can include user profiles with defined access levels 126.The user profiles 126 may additionally or alternately include anapplication program.

The data processing system 116 can include a Trend Analysis Module (thatmay be an application program similar to the modules discussed abovewith respect to FIG. 11) that can access electronically stored studenttest records and underlying cohort data and generate a visualoutput/display of a graph of test trends. A trend can be electronicallygenerated and shown on a display associated with a client 35 (e.g., anadministrator, professor/teacher, or student. The trend can be ingraphic form and may indicate a risk of failure or a need for anintervention or adjustment in a curriculum based at least in part on theresults. The system 10 can be configured to generate a “flag” thatincreases the report frequency if a student (or group of students or aparticular class) is identified as being below minimum. The system 10may also be configured to alert students, advisors, professors/teachersvia email, postal mail and/or using text messages or other suitablecommunication protocol to notify one or more of a negative trend or a“failure” in one or more microcompetencies.

As will be appreciated by those of skill in the art, the operatingsystem 152 may be any operating system suitable for use with a dataprocessing system, such as, but not limited to, those from Microsoft,Inc. (Windows), Apple Computer, Inc. (MacOS), Wind River (VxWorks),RedHat (Linux), LabView or proprietary operating systems. Theinput/output device drivers 158 typically include software routinesaccessed through the operating system 152 by the application programs154 to communicate with devices such as the input/output circuits 146and certain memory 136 components. The application programs 154 areillustrative of the programs that implement various features of thecircuits and modules according to some embodiments of the presentinvention. Finally, the data 156 represents the static and dynamic dataused by the application programs 154, the operating system 152, theinput/output device drivers 158 and other software programs that mayreside in the memory 136.

While the present invention is illustrated with reference to theapplication programs 120, 124, 125 in FIG. 11 as will be appreciated bythose of skill in the art, other configurations fall within the scope ofthe present invention. For example, rather than being applicationprograms 154 these circuits and modules may also be incorporated intothe operating system 152 or other such logical division of the dataprocessing system. Furthermore, while the application programs 120, 124,125 (122) are illustrated as modules in a single data processing system,as will be appreciated by those of skill in the art, such functionalitymay be distributed across one or more data processing systems. Thus, thepresent invention should not be construed as limited to theconfiguration illustrated in FIG. 11 but may be provided by otherarrangements and/or divisions of functions between data processingsystems. For example, although FIG. 11 is illustrated as having variouscircuits and modules, one or more of these circuits or modules may becombined without departing from the scope of the present invention.

Typically, during “on-boarding” or customer set-up, a client 35 isbrought into the network or system 10 and assigned one or more privacylevels based on a legal or organizational entitlement to send and/orreceive certain types (and/or content) of data. An organization mayinclude one or a plurality of web clients 35, each with one or moredifferent assigned privacy levels. The privacy level can define whatdata that entity or person associated with that entity can receive, sendor access.

Brief reference is now made to FIG. 12, which is a screen shot of agraphical user interface for a sub-cohort manager according to someembodiments of the present invention. As illustrated, student groups maybe created major modified by including or excluding specificindividuals. As illustrated, no individuals are shown as being excludedfrom the example sub-cohort. In some embodiments, the student groups maybe used to define the grid rows (FIG. 16, 202) for an interactiveevaluation grid 200.

Reference is now made to FIG. 13, which is a screen shot of a graphicaluser interface for a summative report after grading according to someembodiments of the present invention. Note that the specific postauthors, which are listed in the “Post Author” column are redacted fromthe screen shot. Each line in the report corresponds to a single postmade in a discussion event. For each post, the type of post (e.g.,Student Post Logistics, Student Post Content, Faculty Post Guidance,Student Post Other, etc.), the identifications of may applicablemicrocompentencies, and a corresponding RVU may be listed, in addition,a comment, field is provided to receive and record any comments for eachpost.

Reference is now made to FIG. 14, which is a screen shot of a graphicaluser interface for a post verification report before the data issubmitted to the grid according to some embodiments of the presentinvention. The post verification report may provide a RVU Commit Summaryportion that summarize the students and corresponding RVUs in adiscussion event. Additionally, a Student Posts portion may provide adetailed report of each student in the discussion event and themicrocompetencies and corresponding RVUs earned therein. Additionally, aGroup Overview Portion may provide a detailed report of the group totalmicrocompetencies and corresponding RVUs earned in the discussion event.

Reference is now made to FIG. 15, which is a screen shot of a graphicaluser interface for a managing submitted reports according to someembodiments of the present invention. The user interface may include a“committed” status indicator, and columns for the date, the courseidentifier, the group identifier and the grading summary for each of thecommitted discussion events.

Reference is now made to FIG. 16, which is a partial screen shot of anexemplary interactive evaluation grid 200 that is parsed to display asingle system according to some embodiments of the present invention. Asdiscussed above, as the evaluation grid is interactive, it may be alsoreferenced as a graphical user interface. As illustrated, the parsedgrid represents grid data corresponding to the musculoskeletal system.The student identifiers (Student #) are redacted from the screen, butare understood to be the unique identifiers corresponding to differentstudents, which correspond to rows in the grid. The columns in the gridcorrespond to the sub-topics in and/or related to the musculoskeletalsystem. The total number of points within musculoskeletal system may beprovided as well as high, low and average points corresponding to eachsub-topic.

Reference is now made to FIG. 17, which is a partial screen shot of agraphical user interface for a managing an interactive evaluation gridaccording to some embodiments of the present invention. The grid managermay list each of the grids that are currently defined. In this manner, asingle interface screen may provide selection and access where multiplegrids are presented for editing.

Reference is now made to FIG. 18, which is a partial screen shot of anexemplary interactive evaluation grid that is parsed to analyze the databy discipline according to some embodiments of the present invention. Asillustrated, the parsed grid represents grid data corresponding to theanatomy discipline. The student identifiers (Student #) are redactedfrom the screen, but are understood to be the unique identifierscorresponding to different students, which correspond to rows in thegrid. The columns in the grid correspond to the sub-topics in and/orrelated to the anatomy discipline. The total number of points withinanatomy discipline may be provided as well as high, low and averagepoints corresponding to each sub-topic.

Brief reference is made to FIG. 19, which is a partial screen shot of agraphical user interface according to some embodiments of the presentinvention. Some embodiments of the user interface include a component ofthe interactive evaluation grid that allows selective viewing of one ormore modalities and provides for the definition of the analysis timeinterval. For example, the analysis time interval may be selected byidentifying start and stop times and/or dates of the desired interval.

Brief reference is made to FIG. 20, which is a screen shot of agraphical user interface of a cohort manager according to someembodiments of the present invention. As illustrated, the cohort managermay be used to determine which students and faculty are included in thecohort. For example, student or faculty names may be selected and movedfrom the excluded window to the included window to identify members of acohort.

Reference is now made to FIG. 21, is a partial screen shot of agraphical user interface for a managing an interactive evaluation gridaccording to some embodiments of the present invention. As illustrated,where competencies, such as, for example, “03 Anatomy Discipline” may bedefined by and/or correspond to multiple microcompetencies, which may belisted and displayed in an associated scrollable window.

Reference is now made to FIG. 22, which is a screen shot of a graphicaluser interface illustrating raw imported exam data after an itemanalysis has been performed according to some embodiments of the presentinvention. A didactic exam is a series of binary events that arepresented to test-takers so they can make the binary choice. A learningmanagement system, as disclosed herein is irrelevant to the result ofthe completed exam for a specific cohort. In some embodiments, theresult of an exam is a simple delimited file that includes the followingdata for each test item: unique student identifier; test itemidentifier; microcompetency code; binary choice (0 for incorrect, 1 forcorrect): and relative value unit. For an exam with 100 questions for 10students, the resultant file will have 1000 rows for these four columnsof data. Some embodiments provide that when the file is imported threeadditional items may be added, namely: program identifier (school,university, etc); date of exam; and course identifier. Once the raw datais digested into the data source environment, the raw data is listedexactly as the raw import file as a first check for validation. Asillustrated, the raw imported exam data includes a row for each testitem identifier. In some embodiments, each row may include the name (oridentifier) of the student, the test item identifier (i.e., which examquestion), the associated microcompetency and the earned RVU. Once theraw data is digested into the data source environment, the raw data islisted exactly as the raw import file as a first check for validation. Acomparison with the original delimited file can be done manually orprogrammatically.

Brief reference is made to FIG. 23, which is a partial screen shot of agraphical user interface for checking a raw data report in preparationfor validation according to some embodiments of the present invention. Acomparison to the grading summary, which lists how many rows are in theraw import display may be useful in identifying in under and/or overinclusions of data corresponding to the exam data.

At this point the administrator “commits” the raw data for conversionfrom binary presentation to microcompetency presentation. Each studentis represented with a summary of the points by microcompetency. Multiplequestions in one exam may have the same microcompetency designation. Thetotal number of points (RVUs) is then validated against the originalnumber of points that the learning management system calculated in thebinary presentation. As such, this step converts individual test itemsto topic-associated results.

Brief reference is made to FIG. 24, which is a screen shot of agraphical user interface illustrating an RVU commit summary screenbefore the data is committed according to some embodiments of thepresent invention. Note that before the data is committed, as indicatedby the status “Not Committed”, the RVUs in the RVU Commit Summary aredisplayed as 0.0.

Reference is now made to FIG. 25, which is a partial screen shot of agraphical user interface illustrating data that was collected and mergedby microcompetency code according to some embodiments of the presentinvention. The student scores are displayed for an administrator orother evaluator to commit the data. For each student, a total RVU isindicated, and RVUs for each microcompetency are listed. Someembodiments provide an approval interface, such as, for example a buttonor cheek box. As a separate step, the administrator manually validatesthat the number of RVUs is correct with a check step.

Reference is now made to FIG. 26, which is a screen shot of a graphicaluser interface illustrating data that was collected and merged bymicrocompetency code and that has been verified to provide all studentswith the correct score according to some embodiments of the presentinvention. Note that each of the student check boxes are selectedindicating that the data has been verified. Additionally, referring toFIG. 27, which is a partial screen shot of a graphical user interfaceillustrating the RVU commit summary screen that includes the scoresready to commit to the grid according to some embodiments of the presentinvention, when the data is checked it moves the points to a commitsummary data page for one remaining check.

After validation, each data element is stored to the common datarepository with a unique identifier. The data may be represented in thegrid based on the rules of the specific grid. For example, exam data mayparse in grids where exam data is supposed to be displayed.

One must appreciate the importance of the validation steps. Theresulting grid depends on the correct assignment of microcompetencycodes to exam items and the aggregation of this data for representation.Important decisions about student weakness and strength may be madebased on the grid data. The aggregate data page associates the studentpoints with the specific student. When the administrator is satisfiedthat the data is correct, they will “commit” a grade report to the gridfor display. The import file is tagged as “committed” and the next fileis encountered.

The foregoing is illustrative of the present invention and is not to beconstrued as limiting thereof. Although a few exemplary embodiments ofthis invention have been described, those skilled in the art willreadily appreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention as defined inthe claims. In the claims, means-plus-function clauses, where used, areintended to cover the structures described herein as performing therecited function and not only structural equivalents but also equivalentstructures. Therefore, it is to be understood that the foregoing isillustrative of the present invention and is not to be construed aslimited to the specific embodiments disclosed, and that modifications tothe disclosed embodiments, as well as other embodiments, are intended tobe included within the scope of the appended claims. The invention isdefined by the following claims, with equivalents of the claims to beincluded therein.

That which is claimed is:
 1. A computer program product for operating anelectronic evaluation platform comprising a presentation tier, abusiness logic tier in communication with the presentation tier, and astudent record data tier in communication with the presentation tier andthe business logic tier, the computer program product-comprising anon-transitory computer readable storage medium having computer readableprogram code embodied in the medium that when executed by at least oneprocessor causes the at least one processor to perform operationscomprising: providing within the business logic tier a plurality ofunique microcompetency codes, wherein each unique microcompetency codeof the plurality of unique microcompetency codes categorizes an area ofknowledge and/or skill associated with a performed activity andcomprises a first topic code, a first sub-topic code dependent on thefirst topic code, and a second sub-topic code dependent on the firstsub-topic code; creating, by the business logic tier, a plurality ofunique data records within the student record data tier that areassociated with a plurality of individuals and are automatically updatedover time, respective ones of the plurality of unique data recordscomprising: a unique microcompetency code of the plurality of uniquemicrocompetency codes that is associated with an activity performed by arespective individual of the plurality of individuals; and a relativeeducational value unit (RVU) quantity associated with the activitycomprising a time-based value that assesses a complexity of theactivity; for each of the plurality of individuals, automaticallycalculating, by the business logic tier, a summed RVU quantity of theRVU quantities of the unique data records from the student record datatier that are associated with the respective individual for each of theplurality of unique microcompetency codes; automatically calculating, bythe business logic tier, at least three standard deviations of thesummed RVU quantities of the plurality of individuals for each of theplurality of unique microcompetency codes; generating, by thepresentation tier, a cumulative analysis grid that automaticallyconcurrently obtains or imports the current unique data records from thestudent record data tier and/or calculated by the business logic tierthat are associated with the plurality of individuals and indicatesspecific ones of the plurality of unique microcompetency codes having asummed RVU quantity that is below or above a defined value with respectto the at least three standard deviations calculated by the businesslogic tier; electronically detecting, by the presentation tier, a userselection of a portion of the cumulative analysis grid on a graphicaluser interface of a computer system, wherein content of the graphicaluser interface is provided to the computer system by the presentationtier over a network between the computer system and the presentationtier; and selectively updating, by the presentation tier, the graphicaluser interface of the computer system responsive to the user selectionwithin the graphical user interface of the portion of the cumulativeanalysis grid to reveal supporting unique data records from the studentrecord data tier and/or calculated by the business logic tier forindividuals associated with the selected portion.
 2. The computerprogram product of claim 1, wherein the operations further comprise:automatically generating a cumulative data record comprising the summedRVU quantities of the data records associated with each of the pluralityof individuals for each unique microcompetency code of the plurality ofunique microcompetency codes; modifying the cumulative data record toselectively remove identification information for each of the pluralityof individuals; and automatically providing the cumulative data recordto an accreditation and/or licensing service without human intervention.3. The computer program product of claim 1, wherein the operationsfurther comprise: determining first individuals of the plurality ofindividual commonly having a first summed RVU quantity for the uniquemicrocompetency code that is below the defined value with respect to theat least three standard deviations; and automatically identifying acommon characteristic shared by the first individuals, wherein thecommon characteristic is an instructor, an activity time, a textbook,and/or a location where the activity associated with the uniquemicrocompetency code was performed.
 4. The computer program product ofclaim 3, wherein the operations further comprise automatically alteringthe instructor, the activity time, the textbooks, and/or the locationresponsive to determining the common characteristic.
 5. The computerprogram product of claim 1, wherein the operations further comprise:associating, by the business logic tier, a flag with a first individualof the plurality of individuals responsive to the summed RVU quantityfor the first individual being below the defined value for at least oneof the plurality of unique microcompetency codes, increasing, by thebusiness logic tier, an update frequency of the calculation of thesummed RVU quantity of the at least one of the plurality of uniquemicrocompetency codes for the first individual responsive to a presenceof the flag.
 6. A computer analysis system for automatically trackingstudent competence comprising: a processor; a data repository inelectronic communication with the processor comprising: a first datarecord associated with a first activity performed by a first individualcomprising: a first microcompetency code that categorizes a firstknowledge and/or skill associated with the first activity, the firstmicrocompetency code comprising a first topic code, a first sub-topiccode dependent on the first topic code, and a second sub-topic codedependent on the first sub-topic code; and a first relative educationalvalue unit (RVU) quantity associated with the first activity, the firstRVU quantity comprising a first time-based value that corresponds to acomplexity of the first activity; and a second data record associatedwith a second activity performed by a second individual comprising: thefirst microcompetency code that categorizes the first knowledge and/orskill associated with the second activity; a second RVU quantityassociated with the second activity, the second RVU quantity comprisinga second time-based value that corresponds to a complexity of the secondactivity; and a memory coupled to the processor and comprising computerreadable program code that when executed by the processor causes theprocessor to perform operations comprising: generating a normalizedcomparison between the first activity and the second activity based on adetermination that the first data record and the second data recordcomprise the first microcompetency code; selectively updating a displayof the computer analysis system with a cumulative analysis grid based onresults of the normalized comparison that is correlated by the firsttopic code and the first sub-topic code; identifying a plurality ofunique data records associated with the first individual and a thirdmicrocompetency code, wherein respective ones of the plurality of uniquedata records each comprise a third RVU quantity; calculating a summedRVU quantity of the third RVU quantity of the plurality of unique datarecords for the third microcompetency code; establishing a threshold RVUquantity associated with competency in a second knowledge and/or skillcategorized by the third microcompetency code; updating the cumulativeanalysis grid to identify the first individual responsive to the summedRVU quantity exceeding the threshold RVU quantity; associating a flagwithin the data repository with the first individual responsive to thesummed RVU quantity being below the threshold RVU quantity; andincreasing an update frequency of the normalized comparison of the firstindividual responsive to a presence of the flag, wherein the firstactivity is a first electronic posting on an electronic repository of athreaded discussion, wherein the first microcompetency code isassociated with a topic of the threaded discussion, wherein the firstRVU quantity is determined by keywords, length, and/or discussion timeassociated with the electronic posting, wherein the second activity is asecond electronic posting on the electronic repository of the threadeddiscussion, and wherein the first RVU quantity associated with the firstactivity is different than a second RVU quantity associated with thesecond activity based on a difference in content of the first electronicposting as compared to the second electronic posting.
 7. The computeranalysis system of claim 6, wherein the operations further comprise:summing the first RVU quantity and the second RVU quantity based on thedetermination that the first data record and the second data recordcomprise the first microcompetency code.
 8. The computer analysis systemof claim 6, wherein the first RVU quantity further represents acompetency assessment of a performance of the first activity, such thata higher level of performance of the first activity results in a largerfirst RVU quantity, and a lower level of performance of the firstactivity results in a smaller first RVU quantity.
 9. The computeranalysis system of claim 6, wherein the first RVU quantity is based on apre-defined assessment of difficulty and/or an estimated time tocomplete the first activity.
 10. The computer analysis system of claim6, wherein the operations further comprise: identifying a secondplurality of unique data records associated with a plurality ofindividuals and a fourth microcompetency code, wherein respective onesof the second plurality of unique data records each comprise a fourthRVU quantity; for each of the plurality of individuals, calculating asecond summed RVU quantity of the fourth RVU quantities of data recordsof the second plurality of unique data records that are associated withthe individual for the fourth microcompetency code; calculating at leastthree standard deviations of the second summed RVU quantities of theplurality of individuals; and updating the display of the computeranalysis system to separately identify individuals of the plurality ofindividuals whose second summed RVU quantities are within a firststandard deviation, a second standard deviation, or a third standarddeviation of the second summed RVU quantities of the plurality ofindividuals.
 11. The computer analysis system of claim 6, wherein thefirst activity is a clinical procedure involving clinic time and labtime, and wherein the first RVU quantity is determined by a complexitymultiplier times a sum of the clinic time associated with the firstactivity and the lab time associated with the first activity.
 12. Thecomputer analysis system of claim 6, wherein the first RVU quantity isassigned based on a rubric associated with the first activity.
 13. Thecomputer analysis system of claim 6, wherein the first and second datarecords further comprise a modality indicator that categorizes amodality type of the first and second activity, respectively, whereinthe modality indicator comprises a didactic modality type, an experiencemodality type, and a discussion modality type, and wherein thecumulative analysis grid is further correlated by correlated by didacticmodality, experience modality, and/or discussion modality.
 14. Thecomputer analysis system of claim 6, wherein the first microcompetencycode further comprises a third sub-topic code dependent on the secondsub-topic code.
 15. The computer analysis system of claim 14, whereinthe first topic code, the first sub-topic code, the second sub-topiccode, and the third sub-topic code of the first microcompetency code aredisplayed in the cumulative analysis grid as numbers separated byperiods.
 16. The computer analysis system of claim 6, wherein the firstactivity performed by the first individual is performed during a firstencounter, and wherein the second activity performed by the secondindividual is performed during a second encounter, different from thefirst encounter.
 17. A computer program product for operating anelectronic student competency evaluation platform comprising apresentation tier, a business logic tier in communication with thepresentation tier, and a student record data tier in communication withthe presentation tier and the business logic tier, the computer programproduct-comprising a non-transitory computer readable storage mediumhaving computer readable program code embodied in the medium that whenexecuted by at least one processor causes the at least one processor toperform operations comprising: providing within the business logic tiera plurality of unique microcompetency codes, wherein each uniquemicrocompetency code of the plurality of unique microcompetency codescategorizes an area of knowledge and/or skill and comprises a firsttopic code of a first educational class, a first sub-topic codedependent on the first topic code, and a second sub-topic code dependenton the first sub-topic code; creating, by the business logic tier, aplurality of unique data records within the student record data tier foreach of a plurality of students of an academic cohort of an academicinstitution, each respective one of the plurality of unique data recordscomprising: a didactic relative educational value unit (RVU) quantityassociated with an educational didactic activity electronicallycorrelated to one or more of the unique microcompetency codes andcomprising a first time-based value that assesses a first competency anda first complexity of the educational didactic activity; an experientialRVU quantity associated with an educational experiential activityelectronically correlated to one or more of the unique microcompetencycodes and comprising a second time-based value that assesses a secondcompetency and a second complexity of the educational experientialactivity; and a discussion RVU quantity associated with an educationaldiscussion activity electronically correlated to one or more of theunique microcompetency codes and comprising a third time-based valuethat assesses a third competency and a third complexity of theeducational discussion activity; for each student, calculating, by thebusiness logic tier, a summed RVU quantity of the educational didactic,experiential and discussion RVU quantities from the student record datatier; calculating, by the business logic tier, at least three standarddeviations of the summed RVU quantities of the plurality of students;generating, by the presentation tier, a web interface that comprises acumulative analysis grid to separately identify each student of thestudents as within a first standard deviation, a second standarddeviation, or a third standard deviation calculated by the businesslogic tier of the summed RVU quantities with respect to each of theunique microcompetency codes; providing, by the presentation tier, adisplay of student performance based on the cumulative analysis gridwithin the web interface; accepting, by the business logic tier,additional unique data records over time for each student of theplurality of students for storage into the student record data tier, theadditional unique data records comprising: an additional didactic RVUquantity associated with an additional educational didactic activityelectronically correlated to one or more of the unique microcompetencycodes; an additional experiential RVU quantity associated with anadditional educational experiential activity electronically correlatedto one or more of the unique microcompetency codes; and an additionaldiscussion RVU quantity associated with an additional educationaldiscussion activity electronically correlated to one or more of theunique microcompetency codes; for each student, calculating, by thebusiness logic tier, an additional summed RVU quantity of the additionaleducational didactic, experiential and discussion RVU quantities;calculating, by the business logic tier, at least three standarddeviations of the additional summed RVU quantities of the plurality ofstudents; updating, by the presentation tier, the cumulative analysisgrid within the web interface to separately identify students as withina first standard deviation, a second standard deviation, or a thirdstandard deviation of the additional summed RVU quantities calculated bythe business logic tier with respect to each of the uniquemicrocompetency codes; and selectively updating, by the presentationtier, the display of student performance based on the updated cumulativeanalysis grid within the web interface.
 18. The computer program productof claim 17, wherein the cumulative analysis grid provides rows andcolumns, with the columns providing the summed RVU quantities and therows providing references to ones of the plurality of students, andwherein grid cells of the cumulative analysis grid are color coded basedon the first standard deviation, the second standard deviation, and/orthe third standard deviation of the summed RVU quantities.
 19. Thecomputer program product of claim 17, wherein the operations furthercomprise: establishing a threshold RVU quantity associated withcompetency in a knowledge and/or skill categorized by a firstmicrocompetency code of the unique microcompetency codes; associating,by the business logic tier, a flag with a first student of the pluralityof students responsive to the summed RVU quantity for the first studentbeing below the threshold RVU quantity for the first microcompetencycode; increasing, by the business logic tier, an update frequency of thecalculation of the summed RVU quantity of the educational didactic,experiential, and discussion RVU quantities from the student record datatier for the first student responsive to a presence of the flag.